Elasticsearch性能方案esrally

本文描述使用esrally对自建集群进行测试的方法(esrally还有其他用法不做讨论)

• 1. esrally 介绍

• 2. 测试准备

  • 2.1 document.json

  • 2.2 index.json

  • 2.3 编写track.json

• 3. 命令

1. esrally 介绍

esrally是基于用户视角的对elasticsearch进行测试的工具。

术语：

• rally:汽车拉力赛

• track:赛道,压测方案

• team/car: es instance

• race: 一场比赛

• tournament: 锦标赛

track.json是esrally的压测方案定义文件,包含以下几部分

• indices:索引定义

• templates:indices template，少使用

• corpora:数据集文件定义

• operations:具体操作,可以没有，直接在schedule或者challenge内定义

• schedule:执行操作时的负载

• challenge:区分不同测试场景,比如append和update，便于分开统计

示例:

{
    "version": 2,
    "description": "xxx",
    "indices": [
        {
            "name": "xxxx",
            "body": "xxx你定义的索引格式请求文件 index.json ",
            "types": ["_doc"]
        }
    ],
    "corpora": [
        {
            "name": "xxxx",
            "documents": [
                {
                    "source-file": "es请求文件:目前仅支持bulk格式",
                    "document-count": 100,
                    "uncompressed-bytes": 84553
                }
            ]
        }
    ],
    "challenges":[
        {
            "name":"xxx",
            "schedule": [
                {
                    "operation": {
                        "name": "putdata",
                        "operation-type": "bulk",
                        "bulk-size": 5
                    }
                    "clients": 4,
                    "warmup-iterations": 1000,
                    "iterations": 1000,
                    "target-throughput": 100
                }
            ]
        }
    ]
}

具体字段参考https://esrally.readthedocs.io/en/latest/track.html

教程视频 https://www.bilibili.com/video/av27114309/?spm_id_from=333.788.videocard.0 (某些字段可能已经不适用)

2. 测试准备

测试前需要准备一下文件

• document.json: 测试请求文件

• index.json: 创建索引请求文件

• track.json: 测试case描述文件

2.1 document.json

格式只支持bulk

示例

{"agent": "\"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:44.0) Gecko/20100101 Firefox/44.0\"", "useragent": {"os": "Windows 10", "os_name": "Windows 10", "name": "Firefox"}, "geoip": {"country_name": "Russian Federation", "location": [44.00749999999999, 56.326899999999995]}, "clientip": "93.120.181.0", "referrer": "\"https://www.elastic.co/guide/en/logstash/current/index.html\"", "request": "/favicon-96x96.png", "bytes": 0, "verb": "GET", "response": 304, "httpversion": "1.1", "@timestamp": "2017-07-03T07:51:49.999Z", "message": "93.120.181.0 - - [2017-07-03T07:51:49.999Z] \"GET /favicon-96x96.png HTTP/1.1\" 304 0 \"-\" \"https://www.elastic.co/guide/en/logstash/current/index.html\" \"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:44.0) Gecko/20100101 Firefox/44.0\""}

2.2 index.json

将数据存入es中进行建模,index创建请求参考https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html

将请求写入文件，例如

{
  "settings": {
    "index.number_of_shards": {{number_of_shards | default(5)}},
    "index.number_of_replicas": {{number_of_replicas | default(0)}}
  },
  "mappings": {
    "_doc": {
      "dynamic": "strict",
      "_source": {
        "enabled": {{ source_enabled | default(true) | tojson }}
      },
      "properties": {
        "@timestamp": { "type": "date" },
        "message": { "type": "text", "index": false },
        "agent": { "type": "keyword", "ignore_above": 256 },
        "bytes": { "type": "integer" },
        "clientip": { "type": "ip" },
        "httpversion": { "type": "keyword", "ignore_above": 256 },
        "response": { "type": "short" },
        "verb": { "type": "keyword", "ignore_above": 256 },
        "tags": { "type": "keyword", "ignore_above": 256 },
        "geoip" : {
          "properties" : {
            "country_name" : { "type": "keyword" },
            "location" : { "type": "geo_point" }
          }
        },
        "useragent": {
          "properties": {
            "name": { "type": "keyword", "ignore_above": 256 },
            "os": { "type": "keyword", "ignore_above": 256 },
            "os_name": { "type": "keyword", "ignore_above": 256 }
          }
        },
        "request": {
          "norms": false,
          "type": "text",
          "fields": {
            "keyword": { "ignore_above": 256, "type": "keyword" }
          }
        },
        "referrer": {
          "norms": false,
          "type": "text",
          "fields": {
            "keyword": { "ignore_above": 256, "type": "keyword" }
          }
        }
      }
    }
  }
}

2.3 编写track.json

主要自定义challenge部分

示例：

{
    "version": 2,
    "description": "blc",
    "indices": [
        {
            "name": "eventdata",
            "body": "index.json",
            "types": ["_doc"]
        }
    ],
    "corpora": [
        {
            "name": "eventdata",
            "documents": [
                {
                    "source-file": "eventdata_sample.json",
                    "document-count": 100,
                    "uncompressed-bytes": 84553
                }
            ]
        }
    ],
    "challenges":[
        {
            "name":"default",
            "schedule": [
                {
                    "operation": {
                        "name": "putdata",
                        "operation-type": "bulk",
                        "bulk-size": 5
                    },
                    "clients": 4,
                    "warmup-iterations": 1000,
                    "iterations": 1000,
                    "target-throughput": 100
                },
                {
                    "operation": {
                        "name": "search",
                        "operation-type": "search",
                        "body": {
                            "query": {
                                "match_all": {}
                            }
                        }
                    },
                    "clients": 4,
                    "iterations": 1000,
                    "target-throughput": 100
                }
            ]
        }
    ]
}

3. 命令

查看测试track

esrally list tracks --track-path=xx

运行track

esrally --track-path=xx  \
--target-hosts=127.0.0.1:9192 \
--pipeline=benchmark-only \
--cluster-health=yellow

结果展示

esrally list races

结果对比

esrally compare --baseline=20160518T112057Z --contender=20160518T112341Z