搭建ELK日志分析系统

ELK由ElasticSearch、Logstash和Kiabana三个开源工具组成。官方网站：https://www.elastic.co/products

Elasticsearch是个开源分布式搜索引擎，它的特点有：分布式，零配置，自动发现，索引自动分片，索引副本机制，restful风格接口，多数据源，自动搜索负载等。

Logstash是一个完全开源的工具，他可以对你的日志进行收集、过滤，并将其存储供以后使用（如，搜索）。

Kibana 也是一个开源和免费的工具，它Kibana可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面，可以帮助您汇总、分析和搜索重要数据日志。

接下来使用elasticsearch 5.6.4，kibana5.2.0 ，logstash5.6.3三个版本进行搭建

一、安装elasticsearch

1. 修改配置文件elasticsearch.yml

cluster.name: my-elk
node.name: node-1
#允许外网访问
network.host: 0.0.0.0 
#访问地址端口号，默认为8200
http.port: 9200 

2. 解压安装包后启动elasticsearch出现错误，提示说不能用root用户启动

oot@iZbp12jkgn61msxjvtrypsZ:/opt/elK/elasticsearch-5.6.4# ./bin/elasticsearch
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2020-03-09T02:55:35,488][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-5.6.4.jar:5.6.4]
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) ~[elasticsearch-5.6.4.jar:5.6.4]
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70) ~[elasticsearch-5.6.4.jar:5.6.4]
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) ~[elasticsearch-5.6.4.jar:5.6.4]

3. 新增用户

#新增user用户组
groupadd user

#在user用户下新增bc用户
useradd -g user bc

#为bc用户配置密码
passwd bc

4. 为新增的bc用户配置权限，打开/etc/sudoers配置文件，新增一条bc用户的信息

# User privilege specification
root    ALL=(ALL:ALL) ALL
bc      ALL=(ALL:ALL) ALL

5. 改变elasticsearch文件夹所有者到bc

sudo chown -R bc:user elasticsearch-5.6.4

6. 切换到bc用户，启动elasticsearch
7. 若出现max file descriptors [65535] for elasticsearch process is too low, increase to at least [65536]错误
   则打开vi /etc/security/limits.conf文件，新增两行配置

bc hard nofile 65536
bc soft nofile 65536

8. 若出现max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]错误
   则打开vi /etc/sysctl.conf ，新增配置

vm.max_map_count=655360

并执行命令

sysctl -p

9. 接下来启动elasticsearch

./bin/elasticsearch -d

10. 用curl命令测试后，表示elasticsearch已经成功安装

root@iZbp12jkgn61msxjvtrypsZ:/opt/elk/elasticsearch-5.6.4# curl 127.0.0.1:9200
{
  "name" : "node-1",
  "cluster_name" : "my-elk",
  "cluster_uuid" : "hmggWjhpQJmlLcx3IrWsEQ",
  "version" : {
    "number" : "5.6.4",
    "build_hash" : "8bbedf5",
    "build_date" : "2017-10-31T18:55:38.105Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.1"
  },
  "tagline" : "You Know, for Search"
}

---

二、安装logstash

1. 在config文件夹下 新建logstash.conf文件，写入配置

input {
    file {
        type => "log"
        path => "/opt/logs/*.log"
        start_position => beginning
    }
}
output {
    stdout {
        codec => rubydebug { }
    }
    elasticsearch {
    hosts => "localhost:9200"
    index => "log-%{+YYYY.MM.dd}"
    }

}


iutput {
    stdout {
        codec => rubydebug { }
    }
    elasticsearch {
    hosts => "localhost:9200"
    index => "log-%{+YYYY.MM.dd}"
    }

}

2.启动后报jvm错误,判断可能是服务器内存不够

[1] 29833
root@iZbp12jkgn61msxjvtrypsZ:/opt/elk/logstash-5.6.3# OpenJDK 64-Bit Server VM warning: Option UseParNewGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000d4cc0000, 724828160, 0) failed; error='Not enough space' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 724828160 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /opt/elk/logstash-5.6.3/hs_err_pid29833.log

3. 打开conf下的jvm.options文件修改默认分配的jvm空间把1g改为512m
   4.后台启动命令

./bin/logstash -f ./config/logstash.conf  &

---

三、安装kibana

1. 配置文件

#请求数据指向的elasticsearch服务器
elasticsearch.url: "http://localhost:9200"

#指定本机ip让外部能访问
server.host: 0.0.0.0

# 端口
server.port: 5601

2.启动

./kibana