elastic 冷热数据分离

分析

• 管理es
  节点角色划分、并发shard与线程池设定、冷热数据分离、定期合并segment等

• 索引特点
  当天索引写入和查询都多；以前的索引数据查询少

• 性能问题
  如果存放当天索引的node存放以前的索引，当用户做大跨度查询历史数据时，过多的磁盘IO和CPU消耗造成拖慢node的写入，即当天索引数据的延迟写入。

• 目的
  通过以前的索引数据迁移到其他node节点，隔离存储，减少对实时数据的影响。

冷热数据隔离操作

考虑：node区分冷热、设置模板新索引到热节点、冷索引迁移冷节点

1. 节点冷热区分tag

   # cat elasticsearch.yml    // 配置文件设置tag区分
   node.attr.tag: cold/hot   
   
   # bin/elasticsearch -d -Enode.attr.box_type=hot  //启动设置
   

2. 设置模板新索引到热节点

   template的正则匹配是最长前缀匹配，遇到-截止

   PUT /_template/template_hot_node   
   {
       "index_patterns" : ["a*","b*","c*","d*","e*","f*","g*","h*","i*","j*","k*","l*","m*","n*","o*","p*","q*","r*","s*","t*","u*","v*","w*","x*","y*","z*"],
       "settings" : {
           "number_of_shards" : 5,
            "number_of_replicas" : "1",
            "index.routing.allocation.include.tag" : "hot"
       },
       "aliases" : {
           "alias" : {}
       }
   }
   

   提前将当天的设置到hot node

   PUT /index-xx/_settings  # 索引必须已创建，否则报错 "reason" : "no such index",
   {
     "index.routing.allocation.include.tag": "hot"
   }
   

3. 冷索引迁移冷节点

   #!/bin/bash
   
   es="10.11.100.108:9201"
   yesterday=`date -d '-1days' +%Y.%m.%d`
   important_index="crm-provider-finance|crm-server-localhost-common"
   migration_index=$(curl -s -XGET "http://es.yw.zhenaioa.com:9201/_cat/indices/*-${yesterday}?h=index"  | grep -v '^\.')
   replication_index=$(curl -s -XGET "http://es.yw.zhenaioa.com:9201/_cat/indices/*-${yesterday}?h=index"  | grep -v '^\.' | grep -Ev ${important_index})
   
   function migration_index_to_cold(){
      for i in `echo $1` 
      do
        curl -X PUT "${es}/${i}/_settings" -H 'Content-Type: application/json' -d'{
           "index.routing.allocation.include.tag": "cold"
        }'
      done
   }
   
   function replication_index_to_zero(){
      for i in `echo $1` 
       do
        curl -XPUT -H "Content-Type: application/json" "${es}/${i}/_settings" -d '{
             "number_of_replicas": 0
        }'
       done
   }
   
   migration_index_to_cold "$migration_index"
   replication_index_to_zero "$replication_index"
   
   

• 查看索引信息

  # GET /crm-server-common-2019.05.09/_settings
  {
    "coreuser-monitor-2019.05.09" : {
      "settings" : {
        "index" : {
          "routing" : {
            "allocation" : {
              "include" : {
                "tag" : "hot"     //在热索引
              }
            }
          },
          "number_of_shards" : "5",
          "provided_name" : "coreuser-monitor-2019.05.09",
          "creation_date" : "1557331200967",
          "number_of_replicas" : "0",
          "uuid" : "Ltd2rG5BTS6caUxDhQSVRw",
          "version" : {
            "created" : "6050199"
          }
        }
      }
    }
  }
  

参考

大规模Elasticsearch集群管理心得

elasticsearch如何做冷热数据