ELK --- 合并多行日志（php.log）

说明

系统中经常有一个事件打印多行日志，比如java、php日志，这里需要将这多行日志一个事件合并到一起发送给elasticsearch，使用logstash的Multiline
例如php日志格式为：

[18-Sep-2016 15:55:58]  [pool www] pid 12548
script_filename = /mnt/data/www/mytest/index.php
[0x00007f82321a9688] filemtime() /mnt/data/www/mytest/Session/drivers/Session_files_driver.php:348
[0x00007fffe4c158b0] gc() unknown:0
[0x00007f82321a8f08] session_start() /mnt/data/www/mytest/libraries/Session/Session.php:140
[0x00007f82321a7ea8] __construct() /mnt/data/www/mytest/Session/drivers/Session_files_driver.php:348
[0x00007f82321a6ec8] _ci_init_library() /mnt/data/www/mytest/Session/drivers/Session_files_driver.php:348
[0x00007f82321a6478] _ci_load_stock_library() /mnt/data/www/mytest/Session/drivers/Session_files_driver.php:348
[0x00007f82321a5c10] _ci_load_library() /mnt/data/www/mytest/Session/drivers/Session_files_driver.php:348
[0x00007f82321a0b48] +++ dump failed

[18-Sep-2016 15:55:58]  [pool www] pid 12548
script_filename = /mnt/data/www/mytest/index.php
[0x00007f82321a9688] filemtime() /mnt/data/www/mytest/Session/drivers/Session_files_driver.php:348
[0x00007fffe4c158b0] gc() unknown:0
[0x00007f82321a8f08] session_start() /mnt/data/www/mytest/libraries/Session/Session.php:140
[0x00007f82321a7ea8] __construct() /mnt/data/www/mytest/Session/drivers/Session_files_driver.php:348
[0x00007f82321a6ec8] _ci_init_library() /mnt/data/www/mytest/Session/drivers/Session_files_driver.php:348
[0x00007f82321a6478] _ci_load_stock_library() /mnt/data/www/mytest/Session/drivers/Session_files_driver.php:348
[0x00007f82321a5c10] _ci_load_library() /mnt/data/www/mytest/Session/drivers/Session_files_driver.php:348
[0x00007f82321a0b48] +++ dump failed
......

配置

input {
    file {
        path => "/var/log/php/www.log.slow"
        codec => multiline {
            pattern => "^\[\d{2}-"
            negate => true
            what => "previous"
        }
    }
}

output {
        elasticsearch {
            hosts => "172.16.11.199"
            index => "php-%{+YYYY.MM.dd}"
        }
}

配置解释

• codec 为input的编码插件，来修改日志输入的格式，可以在logstash输入的时候处理不同的数据，而不用再filter中去过滤
• multiline 合并多行数据
• pattern 正则匹配事件中的行
• negate 默认为false，适用于multiline codec 行不匹配pattern选项指定的正则表达式
• what 如果正则表达式匹配，那么事件属于上一个事件还是下一个事件，可以为next和previous

以上综合意思为：
如果该条日志不匹配pattern中的正则，则该条日志属于上一个事件