服务架构

Elasticsearch + Logstash + Kibana

ELK对日志文件数据进行抽取、分析、存储、展示的日志引擎

Docker 安装镜像包

从镜像仓库拉取

docker pull elasticsearch:8.7.0
docker pull kibana:8.7.0
docker pull logstash:8.7.0
docker更改国内mirror，加速
https://blog.csdn.net/just_for_that_moment/article/details/125308103

从其他服务器打包镜像

在它服务器将指定镜像保存成 tar 归档文件,在目标服务器导入镜像包
docker save elasticsearch:8.7.0 > elasticsearch:8.7.0.tar
docker load -i elasticsearch:8.7.0.tar

docker save  elasticsearch:8.7.0  > elasticsearch:8.7.0.tar

docker load -i  elasticsearch:8.7.0.tar

docker save  elasticsearch:8.7.0  > elasticsearch:8.7.0.tar

docker load -i  elasticsearch:8.7.0.tar

查看安装后的镜像包

docker images

Docker-compose

编写docker-compose.yaml

version: '3'
services:
 elasticsearch:
 image: elasticsearch:8.7.0    #elasticsearch镜像包
 container_name: elasticsearch  #定义容器名称
 restart: always  #开机启动，失败也会一直重启
 environment:
 - "cluster.name=elasticsearch" #设置集群名称为elasticsearch
 - "discovery.type=single-node" #以单一节点模式启动
 - "ES_JAVA_OPTS=-Xms512m -Xmx1024m" #设置使用jvm内存大小
 volumes:
 - /data/elk/elasticsearch/plugins:/usr/share/elasticsearch/plugins #插件文件挂载
 - /data/elk/elasticsearch/data:/usr/share/elasticsearch/data #数据文件挂载
 ports:
 - 9200:9200
 kibana:
 image: kibana:8.7.0
 container_name: kibana
 restart: always
 depends_on:
 - elasticsearch #kibana在elasticsearch启动之后再启动
 environment:
 - ELASTICSEARCH_URL=http://elasticsearch:9200 #设置访问elasticsearch的地址
 ports:
 - 5601:5601
 logstash:
 image: logstash:8.7.0
 container_name: logstash
 restart: always
 volumes:
 - /data/elk/logstash/config/:/usr/share/logstash/config/
 - /data/elk/logstash/pipeline/:/usr/share/logstash/pipeline/
 depends_on:
 - elasticsearch #kibana在elasticsearch启动之后再启动
 links:
 - elasticsearch:es #可以用es这个域名访问elasticsearch服务
 ports:
 - 4560:4560

创建logstash挂载目录

mkdir -p /data/elk/logstash/config
mkdir -p /data/elk/logstash/pipeline

创建配置文件

logstash.yml

mkdir -p /data/elk/logstash/config
mkdir -p /data/elk/logstash/pipeline

文件内容

config:
  reload:
    automatic: true
    interval: 3s
xpack:
  management.enabled: false
  monitoring.enabled: false

pipelines.yml

cd /data/elk/logstash/config
touch pipelines.yml
vim pipelines.yml

文件内容(配置多个管道信息，来收集不同的信息)

-pipeline.id: logstash
 path.config: /usr/share/logstash/pipeline/logstash.conf

logstash.conf

cd /data/elk/logstash/pipeline
touch  logstash.conf
vi logstash.conf

文件内容

input {                  #从filebeat取数据，端口与filebeat配置文件一致
  beats {               
    port => 4560         
  }
}
filter {
   if [filetype] == "log_pd_cd"{   #filebeta定义文件类型
     json {
        source => "message"   #日志内容字段
        remove_field => ["log","offset","tags","instance_id"] #移除字段，不需要采集 
    }
}
}
output {                     #输出
    elasticsearch {
        hosts => [ "es:9200" ]   #es同步docker-compose.yml文件
        index => "pad_cd_server" #设置索引 用于kibana匹配索引
    }
}

运行docker 集群

docker-compose up -d

关闭docker集群

docker-compose down

查看创建容器

docker  ps
docker  exec -it -u 0   {container_id}  bash  #进入容器

验证服务

http://{{host}}:5601/    #kibana
http://{{host}}:9200/    #elasticsearch

补充说明

汉化kibana

将kibana容器内的/usr/share/kibana/config/kibana.yml 添加i18n.locale: zh-CN

docker exec -it {container_id} /bin/bash -c "echo "i18n.locale: zh-CN" >> /opt/kibana/config/kibana.yml"
docker restart {container_id}

取消elasticsearch验证

将elasticsearch容器内的 /usr/share/elasticsearch/config/elasticsearch.yml 添加xpack.security.enabled: false

附录

配置文件

elasticsearch.yml

cluster.name: "docker-cluster"
network.host: 0.0.0.0
xpack.security.enabled: false

kibana.yml

/usr/share/elasticsearch/config/elasticsearch.yml

server.host: "0.0.0.0"
server.shutdownTimeout: "5s"
elasticsearch.hosts: [ "http://elasticsearch:9200" ]
monitoring.ui.container.elasticsearch.enabled: true
i18n.locale: zh-CN