Elasticsearch集群部署与数据导入

1部署

1.1 创建用户

所有节点上执行：
1、es启动时需要使用非root用户，所有创建一个elasticsearch 用户：

useradd elasticsearch

2、为elasticsearch用户添加密码：

passwd elasticsearch

3、将elasticsearch 添加到sudoers

elasticsearch ALL=(ALL) NOPASSWD: ALL

4、创建一个elasticsearch 目录

mkdir /home/{elasticsearch,data}

5、给相应的目录添加权限

chown -R elasticsearch:elasticsearch /home/{elasticsearch ,data}

1.2 es部署

切换到elasticsearch 用户安装

1 、安装jdk1.8
2、上传es安装包
3、解压es

tar -zxvf elasticsearch-6.4.0.tar.gz -C /home/elasticsearch

4、修改配置vi /home/elasticsearch/elasticsearch-6.4.0/config/elasticsearch.yml

#集群名称，通过组播的方式通信，通过名称判断属于哪个集群
cluster.name: elasticsearch 
#节点名称，要唯一
node.name: es-1
#数据存放位置
path.data: /home/elasticsearch/data/es/data
#日志存放位置
path.logs: /home/elasticsearch/data/es/logs
#es绑定的ip地址
network.host: 192.168.30.24
#初始化时可进行选举的节点
##discovery.zen.ping.unicast.hosts: ["hadoop01", "hadoop02", "hadoop03"]

5、使用scp拷贝到其他节点

scp -r elasticsearch-6.4.0/ hadoop02:$PWD

scp -r elasticsearch-6.4.0/ hadoop03:$PWD

6、在其他节点上修改es配置，需要修改的有node.name和network.host

7、启动es（/home/elasticsearch/elasticsearch-6.4.0/bin/elasticsearch -h查看帮助文档）

/home/elasticsearch-6.4.0/bin/elasticsearch -d

8、用浏览器访问es所在机器的9200端口http://192.168.20.20:9200/

{ "name" : "es-1", "cluster_name" : "elasticsearch", "cluster_uuid" : "o3BRRetiS-2jP8eoZGjSTQ", "version" : 
{ "number" : "6.4.0", "build_flavor" : "default", "build_type" : "tar", "build_hash" : "595516e", "build_date" :
 "2018-08-17T23:18:47.308994Z", "build_snapshot" : false, "lucene_version" : "7.4.0", "minimum_wire_compatibility_version" :
 "5.6.0", "minimum_index_compatibility_version" : "5.0.0" }, "tagline" : "You Know, for Search" }

---

2 导入数据

2.1 通过logstash导入csv文件

1、上传logstash-6.4.0.tar.gz包，解压。

tar -zxvf logstash-5.6.3.tar.gz -C /home/elasticsearch/

2、修改logstash配置文件

input {
  file {
    path => ["/home/elasticsearch/data.csv"]
    start_position => "beginning"
  }
}
filter {
  csv {
    separator => ","
    columns => ["cluster","storageRate","resourceRrate","object"]
  } 
}
output {
  elasticsearch {
        hosts => ["192.168.20.20:9200"]
        index => "es_data"
  }
}

3、上传csv数据文件

/home/elasticsearch/data.csv

4、执行命令，进行数据导入

/home/elasticsearch/logstash-6.4.0/bin/logstash -f /home/elasticsearch/import.conf

5、页面查看数据(默认显示10条)

http://192.168.20.20:9200/es_data/_search

---

3 可能遇到以下问题

问题一

max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

意思是：elasticsearch用户拥有的内存权限太小，至少需要262144；
解决办法：
切换到root下，

sysctl -w vm.max_map_count=262144 --执行命令；先临时生效
sysctl -a|grep vm.max_map_count --查看结果

vi /etc/sysctl.conf 添加：(重启之后永久生效)
vm.max_map_count=262144

问题二

max file descriptors [4096] for elasticsearch process is too low, increase to at least [6553]
vi /etc/security/limits.conf * soft nofile 65536 * hard nofile 131072 * soft nproc 2048 * hard nproc 4096

解决办法

chmod -R 755 /home/elasticsearch
chown -R elasticsearch.elasticsearch /home/elasticsearch
/home/elasticsearch-6.4.0/bin/elasticsearch -d --重启es即可jps命令查看进程

问题三

you must change the "path.data" setting Logstash could not be started because there is already another instance using
the configured data directory.  If you wish to run multiple instances, you must change the "path.data" setting

使用下面的启动命令

/home/elasticsearch/logstash-6.4.0/bin/logstash -f import.conf --path.data=/home/elasticsearch/
其中，--path.data是指存放数据的路径