es基本命令熟悉

作者: 狂暴踩踩 | 来源:发表于2021-08-20 15:23 被阅读0次

Elastic Search

参考文档

ES官方文档

ES基本使用

ES的mappings使用

ES健康监控

查看所有节点

GET localhost:9200/_cat/nodes

127.0.0.1 12 59 43    cdfhilmrstw * SH-YJ-WAN-PC

查看ES健康状态

GET localhost:9200/_cat/health

1624349783 08:16:23 elasticsearch green 1 1 6 6 0 0 0 0 - 100.0%

查看所有索引

GETlocalhost:9200/_cat/indices

green open .kibana_7.13.2_001              bbhw6jRuQauNJI-6oQQ-uA 1 0 41   12   2.1mb   2.1mb
green open .apm-custom-link                luJO9deHQIKvy2MplPs_Yg 1 0  0    0    208b    208b
green open .kibana-event-log-7.13.2-000001 kvptbxClT0SuH4DNdjg_kw 1 0  1    0   5.6kb   5.6kb
green open .apm-agent-configuration        VcriCYvJQ_-1Pto1o_ZcJw 1 0  0    0    208b    208b
green open .kibana_task_manager_7.13.2_001 GSBzGrJtS-y1QU8PYRxmmw 1 0 10 2098 293.5kb 293.5kb

基本使用

创建一个文档

POST localhost:9200/mall/user/1

{
    "doc": {
        "name": "hinzzz1",
        "age": 18
    }
}

{
    "_index": "mall",
    "_type": "user",
    "_id": "1",
    "_version": 1,
    "result": "created",
    "_shards": {
        "total": 2,
        "successful": 1,
        "failed": 0
    },
    "_seq_no": 0,
    "_primary_term": 1
}

更新一个文档

POST localhost:9200/mall/user/1/_update
POST localhost:9200/mall/user/1
PUT localhost:9200/mall/user/1

_update

不带_update 更新

PUT

有这个文档内容相同则不进行任何操作 version 不变

有这个文档内容不同则更新 version +1

没有则报错

重复更新 ; version+1

有则更新; 没有则新增

{
    "doc": {
        "name": "hinzzz1",
        "age": 18
    }
}

{
    "_index": "mall",
    "_type": "user",
    "_id": "1",
    "_version": 2,
    "result": "updated",
    "_shards": {
        "total": 2,
        "successful": 1,
        "failed": 0
    },
    "_seq_no": 1,
    "_primary_term": 1
}

查询一个文档

GET localhost:9200/mall/user/1

{
    "_index": "mall",
    "_type": "user",
    "_id": "1",
    "_version": 13,
    "_seq_no": 12,
    "_primary_term": 1,
    "found": true,
    "_source": {
        "doc": {
            "name": "hinzzz1",
            "age": 18
        }
    }
}

删除一个文档

DELETE localhost:9200/mall/user/1

{
    "_index": "mall",
    "_type": "user",
    "_id": "1",
    "_version": 14,
    "result": "deleted",
    "_shards": {
        "total": 2,
        "successful": 1,
        "failed": 0
    },
    "_seq_no": 14,
    "_primary_term": 1
}

批量操作

语法:

{action:{metadata}}
{request body }

注: 一般建议1000-5000文档,大小建议 5-15MB , bulk的线程池配置默认是内核数+1

案例:(注: postman中,入参会报错; 但是会正常调用请求)

POST localhost:9200/mall/user/_bulk

{"create":{"_index":"mall","_type":"user"}
{"name":"_bulk_name","age":18}

{"delete":{"_index":"mall","_type":"user","_id":1}}

{
    "took": 14,
    "errors": false,
    "items": [
        {
            "create": {
                "_index": "mall",
                "_type": "user",
                "_id": "x4SlM3oB6QIUoNJhFvbk",
                "_version": 1,
                "result": "created",
                "_shards": {
                    "total": 2,
                    "successful": 1,
                    "failed": 0
                },
                "_seq_no": 15,
                "_primary_term": 1,
                "status": 201
            }
        },
        {
            "delete": {
                "_index": "mall",
                "_type": "user",
                "_id": "1",
                "_version": 1,
                "result": "not_found",
                "_shards": {
                    "total": 2,
                    "successful": 1,
                    "failed": 0
                },
                "_seq_no": 16,
                "_primary_term": 1,
                "status": 404
            }
        }
    ]
}

Query DSL

基本语法:

QUERY_NAME:{
ARGUMENT:VALUE,
ARGUMENT:VALUE,
...
}

简单的组合查询

POST localhost:9200/mall/_search

{
    "query":{
        "match_all":{}
    },
    "from":0,
    "size":5,
    "sort":{
        "age":"desc"
    },
    "_source":["name","age"]
}

{
    "took": 8,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 2,
            "relation": "eq"
        },
        "max_score": null,
        "hits": [
            {
                "_index": "mall",
                "_type": "user",
                "_id": "45",
                "_score": null,
                "_source": {
                    "name": "wlq",
                    "age": 18
                },
                "sort": [
                    18
                ]
            },
            {
                "_index": "mall",
                "_type": "user",
                "_id": "x4SlM3oB6QIUoNJhFvbk",
                "_score": null,
                "_source": {
                    "name": "_bulk_name",
                    "age": 18
                },
                "sort": [
                    18
                ]
            }
        ]
    }
}

查询条件分词匹配_match

POST localhost:9200/mall/_search

match: 基本类型精确匹配

文本类型会根据查询条件进行分词后进行匹配,最终按照评分进行排序

{
    "query":{
        "match":{
            "age":18
        }
    }
}

{
    "took": 1,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 2,
            "relation": "eq"
        },
        "max_score": 1.0,
        "hits": [
            {
                "_index": "mall",
                "_type": "user",
                "_id": "45",
                "_score": 1.0,
                "_source": {
                    "name": "wlq",
                    "age": 18
                }
            },
            {
                "_index": "mall",
                "_type": "user",
                "_id": "x4SlM3oB6QIUoNJhFvbk",
                "_score": 1.0,
                "_source": {
                    "name": "_bulk_name",
                    "age": 18
                }
            }
        ]
    }
}

In查询

POST localhost:9200/mall/_search

{
    "query":{
        "match":{
            "name":"wlq hinzz hogwarts"
        }
    }
}

{
    "took": 2,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 1,
            "relation": "eq"
        },
        "max_score": 0.6931471,
        "hits": [
            {
                "_index": "mall",
                "_type": "user",
                "_id": "45",
                "_score": 0.6931471,
                "_source": {
                    "name": "wlq",
                    "age": 18
                }
            }
        ]
    }
}

查询条件精确匹配 match_phrase

match_phrase 对于查询条件精确匹配,不进行分词

POST localhost:9200/mall/_search

{
    "query":{
        "match":{
            "match_phrase":"wlq hi"
        }
    }
}

{
    "took": 0,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 0,
            "relation": "eq"
        },
        "max_score": null,
        "hits": []
    }
}

匹配字段全部值_keyword

POST localhost:9200/mall/_search

{
    "query":{
        "match_phrase":{
            "name.keyword":"hinzzz wlq"
        }
    }
}

{
    "took": 0,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 0,
            "relation": "eq"
        },
        "max_score": null,
        "hits": []
    }
}

匹配多字段

POST localhost:9200/mall/_search

{
    "query":{
        "multi_match":{
           "query":"sz",
           "fields":[
               "name",
               "address"
           ]
        }
    }
}

{
    "query":{
        "multi_match":{
           "query":"sz",
           "fields":[
               "name",
               "address"
           ]
        }
    }
}

bool复合查询_or

复合语句可以合并,互相嵌套

must: 必须达到must所列举的所有条件

must_not: 必须不匹配

should: 应该满足,相当于or

POST localhost:9200/mall/_search

{
    "query":{
        "bool":{
            "should":[
                {
                    "match":{
                        "name":"wlq he"
                    }
                },
                {
                    "match":{
                        "age":18
                    }
                }
            ]
        }
    }
}

{
    "took": 1,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 2,
            "relation": "eq"
        },
        "max_score": 1.6931472,
        "hits": [
            {
                "_index": "mall",
                "_type": "user",
                "_id": "45",
                "_score": 1.6931472,
                "_source": {
                    "name": "wlq",
                    "age": 18
                }
            },
            {
                "_index": "mall",
                "_type": "user",
                "_id": "x4SlM3oB6QIUoNJhFvbk",
                "_score": 1.0,
                "_source": {
                    "name": "_bulk_name",
                    "age": 18
                }
            }
        ]
    }
}

bool复合查询_filter

filter结果过滤

POST localhost:9200/mall/_search

{
    "query": {
        "bool": {
            "must": [
                {
                    "match": {
                        "name": "wlq"
                    }
                }
            ],
            "filter": {
                "range": {
                    "age": {
                        "gte": 10,
                        "lte": 20
                    }
                }
            }
        }
    }
}

{
    "took": 0,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 1,
            "relation": "eq"
        },
        "max_score": 0.6931471,
        "hits": [
            {
                "_index": "mall",
                "_type": "user",
                "_id": "45",
                "_score": 0.6931471,
                "_source": {
                    "name": "wlq",
                    "age": 18
                }
            }
        ]
    }
}

term 查询

不知道分词器的存在; 使用用于 keyword numeric date
term并不会对查询条件进行分词;但是默认情况下数据会被分词; 因此beijing shenzhen为条件搜索不到保存进去的数据

造数据

POST localhost:9200/test/term/_bulk

{"create":{"_index":"term","_type":"test"}
{"name":"zhaoliu","age":18,"address":"beijing shenzhen"}

{"create":{"_index":"term","_type":"test"}
{"name":"liuqi","age":16,"address":"hangzhou beijing"}

{"create":{"_index":"term","_type":"test"}
{"name":"zhuba","age":20,"address":"guangzhou shenzhen"}

{
    "took": 6,
    "errors": false,
    "items": [
        {
            "create": {
                "_index": "term",
                "_type": "test",
                "_id": "zoTtM3oB6QIUoNJhZPbV",
                "_version": 1,
                "result": "created",
                "_shards": {
                    "total": 2,
                    "successful": 1,
                    "failed": 0
                },
                "_seq_no": 3,
                "_primary_term": 1,
                "status": 201
            }
        },
        {
            "create": {
                "_index": "term",
                "_type": "test",
                "_id": "z4TtM3oB6QIUoNJhZPbV",
                "_version": 1,
                "result": "created",
                "_shards": {
                    "total": 2,
                    "successful": 1,
                    "failed": 0
                },
                "_seq_no": 4,
                "_primary_term": 1,
                "status": 201
            }
        },
        {
            "create": {
                "_index": "term",
                "_type": "test",
                "_id": "0ITtM3oB6QIUoNJhZPbV",
                "_version": 1,
                "result": "created",
                "_shards": {
                    "total": 2,
                    "successful": 1,
                    "failed": 0
                },
                "_seq_no": 5,
                "_primary_term": 1,
                "status": 201
            }
        }
    ]
}

查询

localhost:9200/term/_search

{
    "query": {
        "term": {
            "name": {
                "value": "zhangsan"
            }
        }
    }
}

{
    "took": 0,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 2,
            "relation": "eq"
        },
        "max_score": 1.0296195,
        "hits": [
            {
                "_index": "term",
                "_type": "test",
                "_id": "yYTsM3oB6QIUoNJhR_Yw",
                "_score": 1.0296195,
                "_source": {
                    "name": "zhangsan",
                    "age": 16,
                    "address": "hangzhou"
                }
            },
            {
                "_index": "term",
                "_type": "test",
                "_id": "z4TtM3oB6QIUoNJhZPbV",
                "_score": 1.0296195,
                "_source": {
                    "name": "zhangsan",
                    "age": 16,
                    "address": "hangzhou"
                }
            }
        ]
    }
}

{
    "query": {
        "term": {
            "address": {
                "value": "beijing hangzhou"
            }
        }
    }
}

{
    "took": 0,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 0,
            "relation": "eq"
        },
        "max_score": null,
        "hits": []
    }
}

aggregation聚合

提供聚合能力; 类似mysql中的group by avg max 等聚合函数
语法格式:

"本次聚合的名字,会展示在结果集中":{
    "AGG_TYPE聚合类型(avg,term,terms)" : {}
}

求平均值

POST localhost:9200/mall/_search

{
    "query":{
        "match":{
            "name":"wlq"
        }
    },
    "aggs":{
        "ageAvg":{
            "avg":{
                "field":"age"
            }
        }
    }
}

mappings

定义一个文档(document)以及他所包含的属性如何存储和索引

字段类型

最新mappings-type官方文档

核心类型

字符串类型(String)

text
keyword

数字类型(Numberic)
long
integer
short
double
byte
float
half_float
scaled_float
日期类型(Date)

date

date_nanos 纳秒

如,同时支持三种格式 ; 但是底层通通都是时间戳形式进行存储

{
    "mappings":{
        "_doc":{
            "propertis":{
                "create_date":{
                    "type":"date"
                    "format":"yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
                }
            }
        }
    }
}

布尔类型(boolean)

boolean

二进制类型(Binary)

binary

数组类型(Array)

array

复杂类型

对象类型(Object)

object 用于单json对象

嵌套类型(Nested)

用于json对象数组

"user" : [ 
    {
      "first" : "John",
      "last" :  "Smith"
    },
    {
      "first" : "Alice",
      "last" :  "White"
    }
  ]

会被索引成

"user.first" : [ "alice", "john" ],
"user.last" :  [ "smith", "white" ]

应该改为

"user":{
    "type":"nested"
    "properties":{
        "first":{"type":"text"},
        "last":{"type":"text"}
    }
}

nested查询示例

{
    "query": {
        "nested": {
            "path": "activityDetail",
            "query": {
                "bool": {
                    "must": [
                        {
                            "match": {
                                "activityDetail.activityCode": "activityCode1"
                            }
                        }
                    ]
                }
            }
        }
    },
    "from": 0,
    "size": 10
}

地理类型

地理坐标(Geo-points)
geo_point

使用字符串存储时，顺序是“纬度，经度”,使用数组存储时，顺序是“经度，纬度”。

地理图形(Geo-Shape)
geo_shape 地图的区域; 如多边形

POST http://localhost:9200/geo_demo

{
    "mappings": {
        "properties": {
            "loaction": {
                "type": "geo_point"
            }
        }
    }
}
{
 "acknowledged": true,
 "shards_acknowledged": true,
 "index": "geo_test"
}

point
shape

范围类型

往es里面给出一个范围区间; 可以根据一个字段查询是否在该区间

integer_range
float_range
long_range
double_range
date_range

特殊类型

ip
token_count
percolator
join
alias
completion

设置mappings

PUT localhost:9200/hot_wind

{
    "mappings": {
        "properties": {
            "age": {
                "type": "integer"
            },
            "name": {
                "type": "text"
            },
            "email": {
                "type": "keyword"
            }
        }
    }
}

{
    "acknowledged": true,
    "shards_acknowledged": true,
    "index": "hot_wind"
}

查看mappings

GET localhost:9200/hot_wind

{
    "hot_wind": {
        "aliases": {},
        "mappings": {
            "properties": {
                "age": {
                    "type": "integer"
                },
                "email": {
                    "type": "keyword"
                },
                "name": {
                    "type": "text"
                }
            }
        },
        "settings": {
            "index": {
                "routing": {
                    "allocation": {
                        "include": {
                            "_tier_preference": "data_content"
                        }
                    }
                },
                "number_of_shards": "1",
                "provided_name": "hot_wind",
                "creation_date": "1624420335435",
                "number_of_replicas": "1",
                "uuid": "7KKav6vBSWy4MGe-EK_O1w",
                "version": {
                    "created": "7130299"
                }
            }
        }
    }
}

mappings添加新的字段映射

PUT localhost:9200/hot_wind/_mappings

{
    "properties": {
        "city": {
            "type": "keyword",
            "index":false
        }
    }
}

{
    "acknowledged": true
}

更新mappings

注意映射不可变更; 需要更新只能走数据迁移

基本语法:

POST _reindex 
{
    "source":{
        "index":"source_index"
    },
    "dest":{
        "index":"dest_index"
    }
}

步骤1:建立新索引

创建新的索引

PUT localhost:9200/hot_south

{
    "mappings": {
        "properties": {
            "name": {
                "type": "keyword",
                "index": true
            },
            "age":{
                "type":"long",
                "index":true
            },
            "email":{
                "type":"text",
                "index":false
            },
            "city":{
                "type":"text",
                "index":true
            }
        }
    }
}

{
    "acknowledged": true,
    "shards_acknowledged": true,
    "index": "hot_south"
}

步骤2:数据迁移

POST localhost:9200/_reindex

{
    "source":{
        "index":"hot_wind"
    },
    "dest":{
        "index":"hot_south"
    }
}

{
    "took": 16,
    "timed_out": false,
    "total": 1,
    "updated": 0,
    "created": 1,
    "deleted": 0,
    "batches": 1,
    "version_conflicts": 0,
    "noops": 0,
    "retries": {
        "bulk": 0,
        "search": 0
    },
    "throttled_millis": 0,
    "requests_per_second": -1.0,
    "throttled_until_millis": 0,
    "failures": []
}

步骤3:迁移数据确认

查看数据 : POST localhost:9200/hot_south/_search
查看索引 : GET localhost:9200/hot_south/_mappings

分词

分词分析

POST http://localhost:9200/_analyze

{
"analyzer": "standard",
"text": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
}

{
    "tokens": [
        {
            "token": "the",
            "start_offset": 0,
            "end_offset": 3,
            "type": "<ALPHANUM>",
            "position": 0
        },
        {
            "token": "2",
            "start_offset": 4,
            "end_offset": 5,
            "type": "<NUM>",
            "position": 1
        },
        {
            "token": "quick",
            "start_offset": 6,
            "end_offset": 11,
            "type": "<ALPHANUM>",
            "position": 2
        },
        {
            "token": "brown",
            "start_offset": 12,
            "end_offset": 17,
            "type": "<ALPHANUM>",
            "position": 3
        },
        {
            "token": "foxes",
            "start_offset": 18,
            "end_offset": 23,
            "type": "<ALPHANUM>",
            "position": 4
        },
        {
            "token": "jumped",
            "start_offset": 24,
            "end_offset": 30,
            "type": "<ALPHANUM>",
            "position": 5
        },
        {
            "token": "over",
            "start_offset": 31,
            "end_offset": 35,
            "type": "<ALPHANUM>",
            "position": 6
        },
        {
            "token": "the",
            "start_offset": 36,
            "end_offset": 39,
            "type": "<ALPHANUM>",
            "position": 7
        },
        {
            "token": "lazy",
            "start_offset": 40,
            "end_offset": 44,
            "type": "<ALPHANUM>",
            "position": 8
        },
        {
            "token": "dog's",
            "start_offset": 45,
            "end_offset": 50,
            "type": "<ALPHANUM>",
            "position": 9
        },
        {
            "token": "bone",
            "start_offset": 51,
            "end_offset": 55,
            "type": "<ALPHANUM>",
            "position": 10
        }
    ]
}

ES的使用

field相同说明

不同的索引下面如果field相同, 那么需要在两个索引下定义相同的field定义; 否则会导致一个field有两种类型, lucene性能急剧下降

Java的连接版本

RestHighLevelClient存在版本问题,在使用中碰到版本不一致时,虽然es操作成功; 但是java代码报错

wapper

鉴于RestHighLevelClient版本冲突问题,需要做好封装; 降低由于es服务器升级导致的代码难以修改

重试

老人告诉我,es会存在超时等问题; 需要做好重试

配置

尽量将配置外部化,配置集中,可以通过配置文件快速了解到es的连接配置(这个也是统一的一个标准.不只针对es)

连接池

默认只有30个连接,要注意自定义控制好连接池的大小

更新频繁问题

es不应该存放变化频繁的信息; 但是这个跟业务有关系, 大佬目前在有频繁更新的时候,出现错误:

version conflict, required seqNo [52718252], primary term [1]. current document has seqNo [52760134] and primary term [1]

发生的节点是: 如doc1; 并发更新状态; 由于version每次都会增加; 然后低版本的version会报上面的错误;