一、多字段

字段实现精确

增加一个keyword字段

使用不同的analyzer

不同语言
pinyin字段的搜索
还支持为搜索和索引指定不同的analyzer

二、Exact Values vs Full Text

Exact Values：包括数字/日期/具体一个字符串（例如“Apple Store”）

ES 中的 keyword

全文本，非结构化的文本数据

ES中的text

image.png

Exact Values 不需要被分词

ES为每一个字段创建一个倒排索引
Exact Value在索引时，不需要做特殊的分词处理

image.png
Character Filters
Tokenizer
Token Filter

三、自定义分词

当es自带的分词器无法满足时，可以自定义分词器。通过自组合不同的组件实现

3.1 Character Filters

在Tokenizer之前对文本进行处理，例如增加删除及替换字符。可以配置多个 Character Filters。会影响Tokenizer和offset信息

一些自带的 Character Filters

HTML script -- 去除html标签
Mapping -- 字符串替换
Pattern replace -- 正则匹配替换

3.2 Tokenizer

将原始的文本按照一定的规则，切分为词（term or token）
ES内置的Tokenizers
whitespace / standard / uax_url_email / pattern / keyword / path hierarchy
可以用java 开发组件实现自己的 Tokenizer

3.3 Token Filters

将Tokenizer 输出的单词（term），进行增加，修改，删除
自带的 Token Filters
Lowercase / stop /synonym (添加近义词)

image.png

词的执行过程
1：分析器的大体执行过程
char filter - >tokenizer -> token filter
2：分词的时机
分词在索引时做，也就是数据写入时。目的是创建倒排索引提高搜索速度。写入的原始数据保存在_source中

image.png

PUT logs/_doc/1
{"level":"DEBUG"}

GET /logs/_mapping

POST _analyze
{
  "tokenizer":"keyword",
  "char_filter":["html_strip"],
  "text": "<b>hello world</b>"
}


POST _analyze
{
  "tokenizer":"path_hierarchy",
  "text":"/user/ymruan/a/b/c/d/e"
}

#使用char filter进行替换
POST _analyze
{
  "tokenizer": "standard",
  "char_filter": [
      {
        "type" : "mapping",
        "mappings" : [ "- => _"]
      }
    ],
  "text": "123-456, I-test! test-990 650-555-1234"
}

//char filter 替换表情符号
POST _analyze
{
  "tokenizer": "standard",
  "char_filter": [
      {
        "type" : "mapping",
        "mappings" : [ ":) => happy", ":( => sad"]
      }
    ],
    "text": ["I am felling :)", "Feeling :( today"]
}

// white space and snowball
GET _analyze
{
  "tokenizer": "whitespace",
  "filter": ["stop","snowball"],
  "text": ["The gilrs in China are playing this game!"]
}


// whitespace与stop
GET _analyze
{
  "tokenizer": "whitespace",
  "filter": ["stop","snowball"],
  "text": ["The rain in Spain falls mainly on the plain."]
}


//remove 加入lowercase后，The被当成 stopword删除
GET _analyze
{
  "tokenizer": "whitespace",
  "filter": ["lowercase","stop","snowball"],
  "text": ["The gilrs in China are playing this game!"]
}

//正则表达式
GET _analyze
{
  "tokenizer": "standard",
  "char_filter": [
      {
        "type" : "pattern_replace",
        "pattern" : "http://(.*)",
        "replacement" : "$1"
      }
    ],
    "text" : "http://www.elastic.co"
}

自定义分析器示例

自定义分析器标准格式是：
PUT /my_index
{
    "settings": {
        "analysis": {
            "char_filter": { ... custom character filters ... },//字符过滤器
            "tokenizer": { ... custom tokenizers ... },//分词器
            "filter": { ... custom token filters ... }, //词单元过滤器
            "analyzer": { ... custom analyzers ... }
        }
    }
}
============================实例===========================
PUT /my_index
{
    "settings": {
        "analysis": {
            "char_filter": {
                "&_to_and": {
                    "type": "mapping",
                    "mappings": [ "&=> and "]
            }},
            "filter": {
                "my_stopwords": {
                    "type": "stop",
                    "stopwords": [ "the", "a" ]
            }},
            "analyzer": {
                "my_analyzer": {
                    "type": "custom",
                    "char_filter": [ "html_strip", "&_to_and" ],
                    "tokenizer": "standard",
                    "filter": [ "lowercase", "my_stopwords" ]
            }}
}}}


============================实例===========================
比如自定义好的analyzer名字是my_analyzer,在此索引下的某个新增字段应用此分析器
PUT /my_index/_mapping
{
   "properties":{
        "username":{
             "type":"text",
              "analyzer" : "my_analyzer"
         },
        "password" : {
          "type" : "text"
        }
    
  }
}
=================插入数据====================
PUT /my_index/_doc/1
{
  "username":"The quick & brown fox ",
   "password":"The quick & brown fox "


}
====username采用自定义分析器my_analyzer,password采用默认的standard分析器==
===验证
GET /index_v1/_analyze
{
  "field":"username",
  "text":"The quick & brown fox"
}

GET /index_v1/_analyze
{
  "field":"password",
  "text":"The quick & brown fox"
}
//官网权威指南是真的讲得好，虽然版本太老，Elasticsearch 2.x 版本，一些api已经不适用了，自定义分析器地址：https://www.elastic.co/guide/cn/elasticsearch/guide/cn/custom-analyzers.html