使用ELK收集Docker Swarm日志

作者: 会动的木头疙瘩儿 | 来源:发表于2018-11-21 14:04 被阅读23次

使用ELK收集Docker Swarm日志
docker swarm elk 日志收集 kibana
Docker swarm ELK日志
Mac系统安装ELK
使用elk收集应用日志
Linux学习elk日志收集工具
ELK收集docker日志
docker swarm
日志收集项目
使用logstash作为docker日志驱动收集日志

源自：http://dockone.io/article/2252
https://github.com/deviantony/docker-elk

本文在现有的docker swarm环境上部署

日志收集的流程

Dockerized环境中的典型ELK日志收集流程如下所示：

源自网络

Logstash负责从各种Docker容器和主机中提取日志，这个流程的主要优点是可以更好地用过滤器来解析日志，Logstash将日志转发到Elasticsearch进行索引，Kibana分析和可视化数据。

elasticsearch是一个文档数据库，以mmap的方式管理索引。mmap是一种内存映射文件的方法，即将一个文件或者其它对象映射到进程的地址空间，实现文件磁盘地址和进程虚拟地址空间中一段虚拟地址的一一对映关系。实现这样的映射关系后，进程就可以采用指针的方式，向操作内存一样读写文件。mmap以页为单位实现映射，操作系统对页的的数量有限制，默认的值太小，elasticsearch要求的最小值是262144，小于此值elasticsearch无法启动。分别在三个node上执行以下命令，临时扩大这种限制：

sysctl -w vm.max_map_count=262144

源文的 docker-stack.yml

version: '3.3'

services:

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.4.0
    ports:
      - "9200:9200"
      - "9300:9300"
    configs:
      - source: elastic_config
        target: /usr/share/elasticsearch/config/elasticsearch.yml
    environment:
      ES_JAVA_OPTS: "-Xmx256m -Xms256m"
    networks:
      - elk
    deploy:
      mode: replicated
      replicas: 1

  logstash:
    image: docker.elastic.co/logstash/logstash-oss:6.4.0
    ports:
      - "5000:5000"
      - "9600:9600"
    configs:
      - source: logstash_config
        target: /usr/share/logstash/config/logstash.yml
      - source: logstash_pipeline
        target: /usr/share/logstash/pipeline/logstash.conf
    environment:
      LS_JAVA_OPTS: "-Xmx256m -Xms256m"
    networks:
      - elk
    deploy:
      mode: replicated
      replicas: 1

  kibana:
    image: docker.elastic.co/kibana/kibana-oss:6.4.0
    ports:
      - "5601:5601"
    configs:
      - source: kibana_config
        target: /usr/share/kibana/config/kibana.yml
    networks:
      - elk
    deploy:
      mode: replicated
      replicas: 1

configs:

  elastic_config:
    file: ./elasticsearch/config/elasticsearch.yml
  logstash_config:
    file: ./logstash/config/logstash.yml
  logstash_pipeline:
    file: ./logstash/pipeline/logstash.conf
  kibana_config:
    file: ./kibana/config/kibana.yml

networks:
  elk:
    driver: overlay

从文中获取镜像可能会超时下不到，我自己下载下来，修改tag后上传到私库使用，以上不包括日志收集器，我将使用logpout来收集日志，下面是我修改源文添加的logpout，logpout容器需要限制容器占用的资源，因为在流量高峰期，日志容器会占大量的资源

version: '3.3'

services:

  elasticsearch:
    image: 172.16.10.192:5000/elasticsearch:6.5.0
    ports:
      - "9200:9200"
      - "9300:9300"
    configs:
      - source: elastic_config
        target: /usr/share/elasticsearch/config/elasticsearch.yml
    environment:
      ES_JAVA_OPTS: "-Xmx256m -Xms256m"
    networks:
      - elk
    deploy:
      mode: replicated
      replicas: 1

  logstash:
    image: 172.16.10.192:5000/logstash:6.5.0
    ports:
      - "5000:5000"
      - "9600:9600"
    configs:
      - source: logstash_config
        target: /usr/share/logstash/config/logstash.yml
      - source: logstash_pipeline
        target: /usr/share/logstash/pipeline/logstash.conf
    environment:
      LS_JAVA_OPTS: "-Xmx256m -Xms256m"
    networks:
     - elk
    deploy:
      mode: replicated
      replicas: 1
  
  logspout:
    image: bekt/logspout-logstash
    environment:
      ROUTE_URIS: 'logstash+tcp://logstash:5000'
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock 
    depends_on:
      - logstash
    networks:
      - elk

    deploy:
      mode: global
      restart_policy:
        condition: on-failure
        delay: 30s


  kibana:
    image: 172.16.10.192:5000/kibana:6.5.0
    ports:
      - "5601:5601"
    configs:
      - source: kibana_config
        target: /usr/share/kibana/config/kibana.yml
    networks:
      - elk
    deploy:
      mode: replicated
      replicas: 1

configs:

  elastic_config:
    file: ./elasticsearch/config/elasticsearch.yml
  logstash_config:
    file: ./logstash/config/logstash.yml
  logstash_pipeline:
    file: ./logstash/pipeline/logstash.conf
  kibana_config:
    file: ./kibana/config/kibana.yml

networks:
  elk:
    driver: overlay

我上传到私库的镜像：

REPOSITORY                                      TAG                 IMAGE ID            CREATED             SIZE
172.16.10.192:5000/logstash                     6.5.0               7d4604365acd        11 days ago         702MB
172.16.10.192:5000/kibana                       6.5.0               fcc1f039f61c        11 days ago         727MB
172.16.10.192:5000/elasticsearch                6.5.0               ff171d17e77c        11 days ago         774MB
docker.elastic.co/elasticsearch/elasticsearch   6.5.0               ff171d17e77c        11 days ago         774MB

并修改了 logstash/pipeline/logstash.conf

input {
    tcp {
        port => 5000
    }
}

## Add your filters / logstash plugins configuration here

filter {
    if [docker][image] =~ /logstash/ {
        drop {}
    }
}


output {
    elasticsearch {
        hosts => "elasticsearch:9200"
        index => "logstash-%{host}"
    }
}

input：表示logstash监听在udp的5000端口收集数据。
fileter：过滤器，表示过滤掉image为logstash的容器实例上报上来的数据。
output：表示如何上报过滤后的数据，这里是通过9200端口上报到elasticsearch数据库。

部署到docker swarm中

因为是私库需要加上 --with-registry-auth

docker stack deploy -c docker-stack.yml --with-registry-auth elk

[root@swarm-m docker-elk]# docker stack services elk
ID                  NAME                MODE                REPLICAS            IMAGE                                    PORTS
b2vqn2d78jw9        elk_kibana          replicated          1/1                 172.16.10.192:5000/kibana:6.5.0          *:5601->5601/tcp
bb0ko12i5267        elk_logstash        replicated          1/1                 172.16.10.192:5000/logstash:6.5.0        *:5000->5000/tcp, *:9600->9600/tcp
bw68dn5fe9zp        elk_logspout        global              3/3                 bekt/logspout-logstash:latest            
pv0lfryjmgxo        elk_elasticsearch   replicated          1/1                 172.16.10.192:5000/elasticsearch:6.5.0   *:9200->9200/tcp, *:9300->9300/tcp

部署完成后可以在访问 http://172.16.10.85:5601