ElasticSearch搭建

一、前期准备工作

环境准备一台机器，ip分别为：192.168.2.128
安装jdk1.8
去官网获取最新Elasticsearch包
curl -O -L https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.1.zip
unzip elasticsearch-5.5.1.zip
cd elasticsearch-5.5.1
./bin/elasticsearch 启动

出现三个错误

2017-08-16T01:27:32,668][INFO ][o.e.b.BootstrapChecks    ] [node-1] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
ERROR: [2] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
[2]max number of threads [1024] for user [biligeci] is too low, increase to at least [2048]
[3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2017-08-16T01:27:32,726][INFO ][o.e.n.Node              ] [node-1] stopping ...
[2017-08-16T01:27:35,707][INFO ][o.e.n.Node              ] [node-1] stopped
[2017-08-16T01:27:35,707][INFO ][o.e.n.Node              ] [node-1] closing ...
[2017-08-16T01:27:35,727][INFO ][o.e.n.Node              ] [node-1] closed

问题一：

[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
原因：无法创建本地文件问题,用户最大可创建文件数太小
解决方法：
vim /etc/security/limits.conf
添加如下内容:

elastic soft nofile 65536
elastic hard nofile 65536
elastic soft nproc 2048
elastic hard nproc 2048
elastic soft memlock unlimited
elastic hard memlock unlimited

重新登录才可生效

问题二：

[2]: max number of threads [1024] for user [biligeci] is too low, increase to at least [2048]
原因：无法创建本地线程问题,用户最大可创建线程数太小
解决方法：
vim /etc/security/limits.d/90-nproc.conf
找到如下内容：
soft nproc 1024修改为soft nproc 2048

问题三：

[3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
原因：最大虚拟内存太小
解决方法：切换到root用户下，修改配置文件sysctl.conf
vim /etc/sysctl.conf
添加下面配置：
vm.max_map_count=655360
并执行命令：
sysctl -p

---

另外还需注意一个问题（在日志发现如下内容，这样也会导致启动失败，这一问题困扰了很久）

[2017-06-14T19:19:01,641][INFO][o.e.b.BootstrapChecks] [elk-1] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-06-14T19:19:01,658][ERROR][o.e.b.Bootstrap] [elk-1] node validation exception
[1] bootstrap checks failed
[1]:system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk

解决：修改配置文件，在配置文件添加一项参数
vim /etc/elasticsearch/elasticsearch.yml
bootstrap.system_call_filter: false

==========================================================

二、logstash

去官网获取最新logstash包
curl -L -O https://artifacts.elastic.co/downloads/logstash/logstash-5.5.2.zip
unzip logstash-5.5.2.zip
cd logstash-5.5.2
新建启动配置文件
vi logstash-simple.conf
文件内容：

input {

  kafka {
    bootstrap_servers => "198.218.1.24:9092,198.218.1.26:9092"
    topics => ["logStash"]
  }

}

filter {
  json {
    source => "message"
  }
}

output {

  stdout {
    codec => rubydebug { metadata => true }
  }

  elasticsearch {
    hosts => ["198.218.35.24:9200","198.218.35.25:9200","198.218.35.26:9200"]
    index => "system_log-%{+YYYY.MM.dd}"
    codec => rubydebug { metadata => true}
  }
}

上面输入使用kafka 也可换成redis或者file 都是可以的。
如：

input {
  file {
    path => "/var/log/bootstrap.log"
    type => "system"
    start_position => "beginning"
  }
  file {
    path => "/home/elasticsearch/testlog/logstash-*.log"
    type => "program"
    start_position => "beginning"
  }
}
output {
  elasticsearch {
    hosts =>["198.218.35.24:9200","198.218.35.25:9200","198.218.35.26:9200"]
    index => "system-%{+YYYY.MM.dd}"
  }
  elasticsearch {
    hosts =>["198.218.35.24:9200","198.218.35.25:9200","198.218.35.26:9200"]
    index => "program-%{+YYYY.MM.dd}"
  }
}

输出日志到es里
启动：./bin/logstash -f logstash-simple.conf