ES 性能测试

监控软件

marvel安装
kibana

测试环境

host	私网IP	IP	配置	操作系统	ES角色
NM-WL-ES-01	10.37.30.2	100.66.137.152	4*8c,512G Mem,8T Disk	CentOS7.2	MD
NM-WL-ES-02	10.37.30.3	100.66.137.153	4*8c,512G Mem,8T Disk	CentOS7.2	MD
NM-WL-ES-03	10.37.30.4	100.66.137.154	4*8c,512G Mem,8T Disk	CentOS7.2	MD
NM-WL-ES-04	10.37.30.5	100.66.137.155	4*8c,512G Mem,8T Disk	CentOS7.2	DD
NM-WL-ES-05	10.37.30.6	100.66.137.156	4*8c,512G Mem,8T Disk	CentOS7.2	DD
NM-WL-ES-06	10.37.30.7	100.66.137.157	4*8c,512G Mem,8T Disk	CentOS7.2	DD

每台起两个ES实例， 3个Master, 9个Data

ES 入库效率

8进程, 连接的数据节点有6个
单条数据 888

{
    "T0": "2017-09-07 10:27:01.670",
    "T1": "43",
    "T10": "2",
    "T11": "460-11",
    "T12": "0x0003BD8B",
    "T13": "0x0000",
    "T14": "115.169.119.159",
    "T15": "TOPON.S11.GW0202-B-HW.NJ.JS.NODE.EPC.MNC011.MCC460.3GPPNETWORK.ORG",
    "T16": "460-11",
    "T17": "0x3203",
    "T18": "1",
    "T19": "0xD3C56A04",
    "T2": "64F011033201FFAB",
    "T20": null,
    "T21": "0xFFFF",
    "T22": "255",
    "T23": "0xFFFFFFFF",
    "T24": "0x00C1CECD",
    "T25": "2017-Sep-07 10:26:59",
    "T26": "460-11",
    "T27": "0x3B40",
    "T28": null,
    "T29": "0xFFFF",
    "T3": "0x13707FEF",
    "T30": null,
    "T31": "4294967295",
    "T32": "0xFFFFFFFF",
    "T33": "1970-Jan-01 07:59:59",
    "T34": null,
    "T35": "0xFFFF",
    "T36": "0x0000",
    "T37": "0x0000",
    "T38": "0x0000",
    "T39": "0",
    "T4": "0x95",
    "T40": "00",
    "T41": "0xFFFF",
    "T42": "0xFF",
    "T43": "0xFFFF",
    "T44": "0",
    "T45": "00",
    "T46": "0xFFFF",
    "T47": "0xFF",
    "T48": "0x00005FD8",
    "T49": "0x31",
    "T5": "460110171455497",
    "T50": null,
    "T51": "F070000018",
    "T6": "8618921771001",
    "T7": "3585880708120215",
    "T8": "0x1E",
    "T9": "10"
}

查看template

curl http://100.66.137.157:9201/_template?pretty

Paste_Image.png

time /slview/logpro/bin/EpcFile2ES_Invoke.pl -f 8 -t test_mmelog -v epc -s TaskWLJ  -c '/slview/test/liyr/mmelog/test_mmelog.yml' -lp 1

real 2m33.672s
echo "4072653/153" | bc
26618 /s

ES 索引占磁盘大小

CASE 1

"include_in_all" : false,
字段类型都为string，无分词
无正排索引

源文件大小 3.3G (json)
转为csv 1.77G

sed -i 's/\"//g' ./HuaweiUDN-MM01_20170907105102_000000.csv_007_10
sed -i -E 's/T[0-9]+\://g' ./HuaweiUDN-MM01_20170907105102_000000.csv_007_10
sed -i 's/{//g' ./HuaweiUDN-MM01_20170907105102_000000.csv_007_10
sed -i 's/}//g' ./HuaweiUDN-MM01_20170907105102_000000.csv_007_10

入库后 1.78G

约为 1：1

CASE 2

"include_in_all" : ture,
string 带分词
store true
doc_values true
入库时间 real 2m54.699s
2.84Gi

约为 1: 1.5