elastic 线程池设置解决logstash-429

logstash日志信息

Why am I seeing bulk rejections in my Elasticsearch cluster?

logstash    | [2019-05-06T16:24:17,476][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 429 ({"type"=>"es_rejected_execution_exception", "reason"=>"rejected execution of processing of [833922829][indices:data/write/bulk[s][p]]: request: BulkShardRequest [[crm-provider-permission-2019.05.06][4]] containing [153] requests, target allocation id: x8vav1ixQ_6ILskMY6IsYw, primary term: 1 on EsThreadPoolExecutor[name = node-hot-2/write, queue capacity = 5000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@4958db63[Running, pool size = 17, active threads = 17, queued tasks = 5004, completed tasks = 598685703]]"})

elastic线程池设置

• 查看线程池状态

  # GET /_cat/thread_pool/write?v
  node_name     name  active queue rejected
  node-7        write      0     0        0
  node-master-2 write      0     0        0
  
  # GET /_cat/thread_pool/write?v&h=node_name,name,active,rejected,completed
  node_name     name  active rejected completed
  node-8        write      0     2393 133703661
  node-7        write      0        0  27376561
  
  # curl -s -XGET "http://127.0.0.1:9200/_cat/thread_pool/write?v&h=node_name,name,active,rejected,completed" 
  

• 调整线程池 Increasing the size of the queue in Elasticsearch?

  • es5以后不能动态调整

    PUT _cluster/settings
    {
      "transient": {
        "thread_pool": {
          "bulk": {
            "queue_size": "1000"
          }
        }
      }
    }
    
    不能调整
    "reason": "transient setting [thread_pool.write.queue_size], not dynamically updateable"
    

  • 配置文件修改

    # cat elasticsearch.yml
    thread_pool:
        write:
            size: 32      // 默认是available processors，由17调整到32
            queue_size: 100000   //由4000调整大小为100000
    processors: 32 
    

  • 验证

    查看reject的值
    # curl -s -XGET "http://es.yw.zhenaioa.com:9201/_cat/thread_pool/write?v&h=node_name,name,active,rejected,completed" |grep node-hot
    
    查看logstash日志，无429