ELK集群问题和设置

1.设置集群自动批量创建索引

PUT /_cluster/settings
{
    "persistent" : {
        "action": {
          "auto_create_index": "true"
        }
    }
}

2.设置最大分片数量（默认每个节点1000）

PUT /_cluster/settings
{
  "persistent": {
    "cluster": {
      "max_shards_per_node":100000
    }
  }
}

3.更改默认jvm内存（默认每个节点1G）

image.png
image.png

3.1、通过环境变量设置。(好像没有用)
过ES_JAVA_OPTS设置这些值来完成：
ES_JAVA_OPTS="-Xms2g -Xmx2g" ./bin/elasticsearch 
ES_JAVA_OPTS="-Xms4000m -Xmx4000m" ./bin/elasticsearch
3.2、这可以通过注释掉jvm.options文件中的Xms和Xmx设置并通
建议设置节点内存的一半即可，不超过32G

4.修改logstash的jvm内存

image.png

5.设置刷新间隔refresh_interval

1.做一次ElasticSearch全量更新，此时可以先关闭自动刷新，全量更新完成之后再打开
# 关闭全部index的数据刷新
PUT
{
  "settings": {
    "refresh_interval": -1 
  }
}

# 调整指定index的刷新间隔为1秒
PUT /user
{
  "settings": {
    "refresh_interval": "1s" 
  }
}

6.设置全量更新前副本为0

image.png

#全量更新完设置副本
PUT /index/_settings
{
    "number_of_replicas": 1
}

7.Elasticsearch修改查询返回最大记录条数（默认为1W条）

#设置返回最大记录条数为1000000
PUT /jjd*/_settings
{
    "index": {
        "max_result_window": 1000000
    }
}

image.png

8.分组查询最大聚和桶数

PUT _cluster/settings
{
  "persistent": {
    "search.max_buckets": 200000
  }
}

9.安装logstash 为windows服务
9.1. 在Logstash bin目录下新增一个run.bat文件，输入下面的内容：

@echo off
title logstash
cd C:\ELK\logstash-7.1.0\bin
logstash -f ..\config\logstash.conf
pause

9.2 下载nssm，点击到下载页面。
解压并拷贝nssm-2.24/win64文件下的nssm.exe到Logstash bin目录下。
9.3. 在Logstash bin目录下命令行执行：

nssm install logstash

弹出nssm service installer窗口：

image.png
image.png
image.png

最后，点击“Install service”。

10.线程池设置解决logstash- retrying failed action with response code: 429

logstash    | [2019-05-06T16:24:17,476][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 429 ({"type"=>"es_rejected_execution_exception", "reason"=>"rejected execution of processing of [833922829][indices:data/write/bulk[s][p]]: request: BulkShardRequest [[crm-provider-permission-2019.05.06][4]] containing [153] requests, target allocation id: x8vav1ixQ_6ILskMY6IsYw, primary term: 1 on EsThreadPoolExecutor[name = node-hot-2/write, queue capacity = 5000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@4958db63[Running, pool size = 17, active threads = 17, queued tasks = 5004, completed tasks = 598685703]]"})

修改yml文件

# cat elasticsearch.yml
thread_pool:
    write:
        size: 32      // 默认是available processors，由17调整到32
        queue_size: 100000   //由4000调整大小为100000
processors: 32 

参考文章https://www.jianshu.com/p/42fe436cd06f

11.处理failed shard on node [xxxxxx]: failed recovery报错

该原因是：某节点上的分片尝试恢复5次没有成功，然后就丢弃不管。导致该分片无法恢复。

POST /_cluster/reroute?retry_failed=true

12.磁盘空间不足导致报错 把索引设置成只读

错误: retrying failed action with response code: 403

//索引
PUT /_all/_settings
{
“index.blocks.read_only_allow_delete”: null
}

13.[parent] Data too large, data for [<http_request>] would be [33444183832/31.1gb], which is larger than the limit of [32534043033/30.2gb],详细错误

{"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [33444183832/31.1gb], which is larger than the limit of [32534043033/30.2gb], real usage: [33444183832/31.1gb], new bytes reserved: [0/0b]","bytes_wanted":33444183832,"bytes_limit":32534043033,"durability":"TRANSIENT"}],"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [33444183832/31.1gb], which is larger than the limit of [32534043033/30.2gb], real usage: [33444183832/31.1gb], new bytes reserved: [0/0b]","bytes_wanted":33444183832,"bytes_limit":32534043033,"durability":"TRANSIENT"},"status":429}

解决方案

vim  /config/elasticsearch.yml

#添加下面配置

# 缓存回收大小，无默认值
# 有了这个设置，最久未使用（LRU）的 fielddata 会被回收为新数据腾出空间
# 控制fielddata允许内存大小，达到HEAP 20% 自动清理旧cache
indices.fielddata.cache.size: 20%
indices.breaker.total.use_real_memory: false
# fielddata 断路器默认设置堆的 60% 作为 fielddata 大小的上限。
indices.breaker.fielddata.limit: 40%
# request 断路器估算需要完成其他请求部分的结构大小，例如创建一个聚合桶，默认限制是堆内存的 40%。
indices.breaker.request.limit: 40%
# total 揉合 request 和 fielddata 断路器保证两者组合起来不会使用超过堆内存的 70%(默认值)。
indices.breaker.total.limit: 95%

或者：

PUT /_cluster/settings
{
  "persistent": {
    "indices.breaker.fielddata.limit": "60%"
  }
} 
 
 
PUT /_cluster/settings
{
  "persistent": {
    "indices.breaker.request.limit": "40%"
  }
} 
 
 
PUT /_cluster/settings
{
  "persistent": {
    "indices.breaker.total.limit": "70%"
  }
}

参考：
1.https://blog.csdn.net/qq_25646191/article/details/108862795
2.https://blog.csdn.net/weixin_42166515/article/details/89093832
3.ES官网限制内存使用：https://www.elastic.co/guide/cn/elasticsearch/guide/current/_limiting_memory_usage.html

14.系统索引相关设置

默认情况下，X-Pack监控客户端会每隔10s采集一次集群的监控信息，并保存到对应ES实例的以.monitoring-*为前缀的索引中。

目前主要有.monitoring-es-6-和.monitoring-kibana-6-这两种索引，以天为单位滚动创建。采集完的信息会保存在以.monitoring-es-6-为前缀，以当前日期为后缀的索引中。

其中.monitoring-es-6-*索引占用磁盘空间较大，主要存放了集群状态、集群统计、节点统计、索引统计等信息。

#设置监控索引的保留天数
PUT _cluster/settings
{
  "persistent": {
        "xpack.monitoring.history.duration":"1d"
  }
}

#设置需要采集的监控索引。
#通过调用API设置哪些索引需要监控以及哪些索引不需要监控，以减少.monitoring-es-6-*索引所占用的磁盘空间。以下命令以禁掉采集系统索引为例
PUT _cluster/settings
{"persistent": {"xpack.monitoring.collection.indices": "*,-.*"}}

15.