九、实战之ELK

ELK 是一整套实时日志处理的解决方案，是三个软件产品的首字母缩写，Elasticsearch，Logstash 和 Kibana。

• Elasticsearch: 存储各类日志
• Logstash: logstash server端用来搜集日志
• Kibana: web化接口用作查寻和可视化日志
  这三款软件都是开源软件，通常是配合使用，而且又先后归于 Elastic.co 公司名下，故被简称为 ELK 协议栈，见下图。

ELK

1. 安装elasticsearch

请参加之前的文章，elasticsearch安装

2. 安装logstash

• 下载logstash 5.6.4

• 修改配置文件jvm.options，默认需要1G，可以根据需要修改

-Xms256m
-Xmx256m

• 测试1（控制台输入，控制台输出）

./logstash -e'input { stdin { } } output { stdout {} }'

• 测试2（控制台输入，输出到ESs）

./logstash -e 'input { stdin { } } output { elasticsearch { hosts => ["127.0.0.1:9200"] } stdout { codec => rubydebug }}'

• 创建配置文件log.conf

input {
    file {
        path => ["/Users/makun/software/elk/logstash-5.6.5/logs/logstash-plain.log"]
    }   
}
output {
    elasticsearch {
        hosts => "127.0.0.1:9200"
        index => "logstash-logs-%{+YYYY.MM.dd}"
        template_overwrite => true
    }
}

• 启动

MacBook-Pro:bin makun$ ./logstash -f ../mk_conf/log.conf 
Sending Logstash's logs to /Users/makun/software/elk/logstash-5.6.5/logs which is now configured via log4j2.properties
[2017-12-09T00:39:34,838][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/Users/makun/software/elk/logstash-5.6.5/modules/fb_apache/configuration"}
[2017-12-09T00:39:34,844][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/Users/makun/software/elk/logstash-5.6.5/modules/netflow/configuration"}
[2017-12-09T00:39:35,473][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://127.0.0.1:9200/]}}
[2017-12-09T00:39:35,475][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://127.0.0.1:9200/, :path=>"/"}
[2017-12-09T00:39:35,628][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://127.0.0.1:9200/"}
[2017-12-09T00:39:35,686][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2017-12-09T00:39:35,693][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2017-12-09T00:39:35,708][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/logstash
[2017-12-09T00:39:35,764][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//127.0.0.1:9200"]}
[2017-12-09T00:39:35,775][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2017-12-09T00:39:40,987][INFO ][logstash.pipeline        ] Pipeline main started
[2017-12-09T00:39:41,069][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

image.png

3. 安装kibana

• 下载kibana 5.6.4
• 修改配置文件kibana.yml

server.port: 5601
server.host: "localhost"
elasticsearch.url: "http://localhost:9200"
kibana.index: ".kibana"

• 启动

MacBook-Pro:bin makun$ ./kibana
  log   [02:41:35.307] [info][status][plugin:kibana@5.6.4] Status changed from uninitialized to green - Ready
  log   [02:41:35.415] [info][status][plugin:elasticsearch@5.6.4] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [02:41:35.456] [info][status][plugin:console@5.6.4] Status changed from uninitialized to green - Ready
  log   [02:41:35.517] [info][status][plugin:metrics@5.6.4] Status changed from uninitialized to green - Ready
  log   [02:41:35.761] [info][status][plugin:timelion@5.6.4] Status changed from uninitialized to green - Ready
  log   [02:41:35.766] [info][listening] Server running at http://localhost:5601
  log   [02:41:35.768] [info][status][ui settings] Status changed from uninitialized to yellow - Elasticsearch plugin is yellow
  log   [02:41:35.995] [info][status][plugin:elasticsearch@5.6.4] Status changed from yellow to green - Kibana index ready
  log   [02:41:35.996] [info][status][ui settings] Status changed from yellow to green - Ready

• 打开 http://localhost:5601/

image.png