日志聚合解决的主要痛点为分布式架构下，查看散落各处日志的不便。
笔者的另一篇博客《ELK on docker-compose》已介绍了搭建一个ELK，并给出了一个docker-compose.yml
无论容器场景还是非容器场景，日志聚合均已十分成熟，可搜到很多材料。本文列举几种docker下日志聚合的方案并做个简单比较，没有关注Beats这种通用工具，更关注利用docker的log-driver机制对接ELK，不需要程序输出日志到文件，而是输出到标准输出stdout和标准错误stderr。

dcoker中几种日志聚合方案简述

Beats系列当然可以使用，和不使用docker时方案没有太大出入，建议将其封入基础镜像；利用docker的log-driver机制，没必要将日志输出到文件，因为默认的机制将程序的标准输出stdout和标准错误stderr持久化了，而且可以通过log-driver机制将日志接入ELK。
具体来说，容器默认的log-driver为json-file，位置/var/lib/docker/containers/<容器ID>/<容器ID>-json.log

$ docker info | grep 'Logging Driver'
Logging Driver: json-file

而本文关注的几种方案利用其他log-driver，几种方案的简介、对比如下：

候选工具	简介
syslog	容器log-driver=syslog，容器的stdout和stderr先转发给linux宿主机上的rsyslog，rsyslog再转发给logstash；syslog需要修改linux中的配置，不便于实践everything as code
gelf	容器log-driver=syslog，再指定要发送给logstash的gelf
fluentd-pilot	理念很有趣，启动一个fluentd-pilot容器，容器不需要指定日志发送的位置，而是通过--label指定特殊标签，github上不够活跃

gelf方案

修改logstash的配置，

docker run -p 12201:12201/udp --link af64b395a5c1:elasticsearch docker.elastic.co/logstash/logstash:6.7.1 logstash -e 'input { gelf{} }  output { elasticsearch {  hosts => ["elasticsearch:9200"] } }'

主要为input部分使用gelf

input {
  gelf{}
}

日志为

$ docker run -p 12201:12201/udp --link af64b395a5c1:elasticsearch docker.elastic.co/logstash/logstash:6.7.1 logstash -e 'input { gelf{} }  output { elasticsearch {  hosts => ["elasticsearch:9200"] } }'
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2019-04-08T01:27:17,093][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[2019-04-08T01:27:17,108][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[2019-04-08T01:27:17,116][WARN ][logstash.runner          ] Deprecated setting `xpack.monitoring.elasticsearch.url` please use `xpack.monitoring.elasticsearch.hosts`
[2019-04-08T01:27:17,500][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-04-08T01:27:17,516][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.7.1"}
[2019-04-08T01:27:17,566][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"4a577feb-363e-4a1b-b96a-4eb4bd528c44", :path=>"/usr/share/logstash/data/uuid"}
[2019-04-08T01:27:18,129][WARN ][logstash.monitoringextension.pipelineregisterhook] xpack.monitoring.enabled has not been defined, but found elasticsearch configuration. Please explicitly set `xpack.monitoring.enabled: true` in logstash.yml
[2019-04-08T01:27:19,020][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
[2019-04-08T01:27:19,195][WARN ][logstash.licensechecker.licensereader] Restored connection to ES instance {:url=>"http://elasticsearch:9200/"}
[2019-04-08T01:27:19,280][INFO ][logstash.licensechecker.licensereader] ES Output version determined {:es_version=>6}
[2019-04-08T01:27:19,288][WARN ][logstash.licensechecker.licensereader] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2019-04-08T01:27:19,437][INFO ][logstash.monitoring.internalpipelinesource] Monitoring License OK
[2019-04-08T01:27:19,439][INFO ][logstash.monitoring.internalpipelinesource] Validated license for monitoring. Enabling monitoring pipeline.
[2019-04-08T01:27:23,298][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-04-08T01:27:23,352][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
[2019-04-08T01:27:23,419][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://elasticsearch:9200/"}
[2019-04-08T01:27:23,431][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-04-08T01:27:23,432][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2019-04-08T01:27:23,456][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//elasticsearch:9200"]}
[2019-04-08T01:27:23,465][INFO ][logstash.outputs.elasticsearch] Using default mapping template
[2019-04-08T01:27:23,490][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2019-04-08T01:27:23,518][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/logstash
[2019-04-08T01:27:23,549][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x3e3ac2e9 run>"}
[2019-04-08T01:27:23,565][INFO ][logstash.inputs.gelf     ] Starting gelf listener (udp) ... {:address=>"0.0.0.0:12201"}
[2019-04-08T01:27:23,627][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-04-08T01:27:23,933][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "document_type" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Document types are being deprecated in Elasticsearch 6.0, and removed entirely in 7.0. You should avoid this feature If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"document_type", :plugin=><LogStash::Outputs::ElasticSearch bulk_path=>"/_xpack/monitoring/_bulk?system_id=logstash&system_api_version=6&interval=1s", hosts=>[http://elasticsearch:9200], sniffing=>false, manage_template=>false, id=>"79d6d565e50fe66f5b522da92c3f148f42013ca30f089ab756df0e5573e82c9c", document_type=>"%{[@metadata][document_type]}", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_6869d230-c856-4978-8174-4a45318a9c76", enable_metric=>true, charset=>"UTF-8">, workers=>1, template_name=>"logstash", template_overwrite=>false, doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_initial_interval=>2, retry_max_interval=>64, retry_on_conflict=>1, ilm_enabled=>false, ilm_rollover_alias=>"logstash", ilm_pattern=>"{now/d}-000001", ilm_policy=>"logstash-policy", action=>"index", ssl_certificate_verification=>true, sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false>}
[2019-04-08T01:27:23,940][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50}
[2019-04-08T01:27:23,955][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
[2019-04-08T01:27:23,961][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://elasticsearch:9200/"}
[2019-04-08T01:27:23,966][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-04-08T01:27:23,966][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2019-04-08T01:27:23,969][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://elasticsearch:9200"]}
[2019-04-08T01:27:24,003][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>".monitoring-logstash", :thread=>"#<Thread:0x3a25e8f1 sleep>"}
[2019-04-08T01:27:24,011][INFO ][logstash.agent           ] Pipelines running {:count=>2, :running_pipelines=>[:".monitoring-logstash", :main], :non_running_pipelines=>[]}
[2019-04-08T01:27:24,284][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

此时，随便启动一个容器，将其log指向此处

docker run --log-driver gelf --log-opt gelf-address=udp://<换用宿主机ip>:12201 --rm alpine echo hello world

注意其中的--log-driver gelf --log-opt gelf-address=udp://<换用宿主机ip>:12201
在kibana即可看到收集效果

利用gelf做日志聚合

elasticsearch收到的json形如：

{
  "_index": "logstash-2019.04.09",
  "_type": "doc",
  "_id": "IfvVAGoBIccTKo5H-R20",
  "_version": 1,
  "_score": null,
  "_source": {
    "@version": "1",
    "container_name": "compassionate_wilson",
    "tag": "b5711ba0d851",
    "message": "hello world",
    "@timestamp": "2019-04-09T06:43:03.692Z",
    "command": "echo hello world",
    "created": "2019-04-09T06:43:03.113896143Z",
    "source_host": "1.203.181.118",
    "version": "1.1",
    "image_id": "sha256:caf27325b298a6730837023a8a342699c8b7b388b8d878966b064a1320043019",
    "image_name": "alpine",
    "container_id": "b5711ba0d8519f8a95a8335bc84c27bb87324dcbcb1f43285a7d80644797231b",
    "level": 6,
    "host": "dk-mi13"
  },
  "fields": {
    "created": [
      "2019-04-09T06:43:03.113Z"
    ],
    "@timestamp": [
      "2019-04-09T06:43:03.692Z"
    ]
  },
  "sort": [
    1554792183692
  ]
}

fluentd-pilot方案

使用fluentd-pilot，可接elasticsearch，启动命令如下：

$ docker run --rm -it \
     -v /var/run/docker.sock:/var/run/docker.sock \
     -v /etc/localtime:/etc/localtime \
     -v /:/host:ro \
     --cap-add SYS_ADMIN \
     -e LOGGING_OUTPUT=elasticsearch \
     -e ELASTICSEARCH_HOST=1.203.181.43 \
     -e ELASTICSEARCH_PORT=9200 \
     registry.cn-hangzhou.aliyuncs.com/acs/log-pilot:0.9.6-fluentd

此时，同样启动一个示例应用，命令如下：

docker run --rm -p 10080:8080 \
--label aliyun.logs.catalina=stdout \
tomcat:9.0.17-jre11

elasticsearch收到的json形如：

{
  "_index": "catalina-2019.04.09",
  "_type": "fluentd",
  "_id": "IPvTAGoBIccTKo5Hdh2P",
  "_version": 1,
  "_score": null,
  "_source": {
    "log": "09-Apr-2019 06:39:19.879 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [684] milliseconds\n",
    "stream": "stderr",
    "time": "2019-04-09T06:39:19.880086298Z",
    "host": "cb3e36b53aec",
    "index": "catalina",
    "topic": "catalina",
    "docker_container": "sad_gagarin"
  },
  "fields": {
    "time": [
      "2019-04-09T06:39:19.880Z"
    ]
  },
  "sort": [
    1554791959880
  ]
}

上述命令试验成功，下述命令失败：

docker run --label aliyun.logs.some-index=stdout --rm alpine echo hello world

发现一点，查到的日志均是stderr里过来的

查到的日志均来自stderr
对比发现，对于tomcat，fluentd-pilot方式下收集到的日志和gelf一样

docker-compose交付基础设施

根据上述内容，新建文件docker-compose.yml，内容如下：

version: '2'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.7.1
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      discovery.type: single-node
  kibana:
    image: docker.elastic.co/kibana/kibana:6.7.1
    depends_on:
      - elasticsearch
    ports:
      - "5601:5601"
  logstash:
    image: docker.elastic.co/logstash/logstash:6.7.1
    depends_on:
      - elasticsearch
    ports:
      - "9600:9600"
      - "12201:12201/udp"
    entrypoint: logstash -e 'input { gelf{} }  output { elasticsearch {  hosts => ["elasticsearch:9200"] } }'
  fluentd-pilot:
    image: registry.cn-hangzhou.aliyuncs.com/acs/log-pilot:0.9.6-fluentd
    depends_on:
      - elasticsearch
    cap_add:
      - SYS_ADMIN
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /:/host:ro
    environment:
      FLUENTD_OUTPUT: elasticsearch
      ELASTICSEARCH_HOST: elasticsearch
      ELASTICSEARCH_PORT: '9200'

在同一层目录下执行如下命令

docker-compose up