LogStach + kibana

作者: 慕知 | 来源:发表于2021-05-09 20:36 被阅读0次

Logstash 是一个开源的数据收集引擎，可以水平伸缩，而且 logstash 整个 ELK 当中拥有最多插件的一个组件，其可以接收来自不同来源的数据并统一输出到指定的且可以是多个不同目的地。

一，LogStach

1，部署

#两台都要下载

1下载安装包
[root@\ es01~]# wget https://artifacts.elastic.co/downloads/logstash/logstash-7.12.1-x86_64.rpm


2，安装
[root@\ es01~]#  yum install logstash-7.12.1-x86_64.rpm -y



3，对数据目录设置权限
[root@\ es02~]# chown -R logstash.logstash /usr/share/logstash/

2,输出

1，测试:输入输出到shell控制台
[root@\ es01~]# /usr/share/logstash/bin/logstash -e 'input { stdin{} } output { stdout{ codec => rubydebug }}'
...
你好
{
          "host" => "es01",
      "@version" => "1",
       "message" => "你好",
    "@timestamp" => 2021-05-09T11:20:19.748Z






2，测试: logstach输出到文件当中
[root@\ es01~]# /usr/share/logstash/bin/logstash -e 'input { stdin{} } output { file { path => "/tmp/log-%{+YYYY.MM.dd}-messages.log"}}'
...
nihao
[INFO ] 2021-05-09 19:24:26.011 [[main]>worker0] file - Opening file {:path=>"/tmp/log-2021.05.09-messages.log"}
[INFO ] 2021-05-09 19:24:38.963 [[main]>worker0] file - Closing file /tmp/log-2021.05.09-messages.log



到文件路径下去查看
[root@\ es01~]# cd /tmp/

[root@\ es01/tmp]# ll
-rw-r--r-- 1 root root 89 May  9 19:24 log-2021.05.09-messages.log

[root@\ es01/tmp]# cat log-2021.05.09-messages.log 
{"host":"es01","message":"nihao","@timestamp":"2021-05-09T11:24:25.726Z","@version":"1"}






3，测试：logstach输出到elasticsearch当中
[root@\ es01~]#  /usr/share/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch {hosts => ["172.16.1.70:9200"] index => "mytest-%{+YYYY.MM.dd}" }}'
...
hello egon


# 查看验证（见图1，图2）






4.测试：logstach输出到redis当中
# redis下载
[root@\ es01~]# yum install -y redis

[root@\ es01~]# vim /etc/redis.conf 
bind 0.0.0.0

[root@\ es01~]# systemctl restart redis



# 测试
[root@es-01 /opt]# /usr/share/logstash/bin/logstash -e 'input { stdin{} } output {redis {  host => "172.16.1.71" port => "6379" data_type => "list" key => "logstash-%{type}" }}'
...
你好 egon


# 验证
[root@\ es01~]# redis-cli --raw
127.0.0.1:6379> keys *
logstash-%{type}

127.0.0.1:6379> LRANGE logstash-%{type} 0 -1
{"@timestamp":"2021-05-09T11:40:53.271Z","@version":"1","message":"你好 egon","host":"es01"}

图1

图2

3，读取

1,logstach读取日志文件
[root@\ es01~]# /usr/share/logstash/bin/logstash -e 'input { file { path => "/var/log/messages" } } output { elasticsearch {hosts => ["172.16.1.70:9200"] index => "system-log-%{+YYYY.MM.dd}" }}'

见图3 图4




2，在标准输出中读取
[root@es-01 /opt]# /usr/share/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch {hosts => ["172.16.1.70:9200"] index => "system-stdin-%{+YYYY.MM.dd}" }}'
...
你好 雪人



见图5 图6

图3

图4

图5

图6

4，分类

1，从多个文件中读取文件
path => "/var/log/messages" #日志路径
type => "systemlog" #事件的唯一类型
start_position => "beginning" #第一次收集日志的位置
stat_interval => "3" #日志收集的间隔时间

[root@es-01 /opt]# /usr/share/logstash/bin/logstash -e 'input { file{ path => "/var/log/messages" type => "systemlog" start_position => "beginning" stat_interval => "3" } file{ path => "/var/log/cron" type => "systemcron" start_position => "beginning" stat_interval => "3" } } output { elasticsearch {hosts => ["172.16.1.70:9200"] index => "system-stdin-%{+YYYY.MM.dd}" }}'

验证见图7




2，分类输出多个数据仓库
[root@\ es01~]# /usr/share/logstash/bin/logstash -e 'input { file{ path => "/var/log/messages" type => "systemlog" start_position => "beginning" stat_interval => "3" } file{ path => "/var/log/cron" type => "systemcron" start_position => "beginning" stat_interval => "3" } } output { if [type] == "systemlog" { elasticsearch {hosts => ["172.16.1.70:9200"] index => "system-systemlog-%{+YYYY.MM.dd}" }} if [type] == "systemcron" { elasticsearch {hosts => ["172.16.1.70:9200"] index => "system-systemcron-%{+YYYY.MM.dd}" } } }'

[root@\ es01~]# echo "cron" > /var/log/cron
验证见图8  图9





3，测试配置文件
[root@\ es01~]# vim test.conf
...
[INFO ] 2021-05-09 20:11:55.316 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

（把上面单引号的内容放在一个文件里）

# 测试配置文件 -t
[root@es-01 ~]# /usr/share/logstash/bin/logstash -f test.conf -t


出现ok字样表示成功



# 使用配置文件
[root@es-01 ~]# /usr/share/logstash/bin/logstash -f test.conf

[root@\ es01~]# echo "good night" >> /var/log/cron 

验证见 图10

图7

图8

图9

图10

二，kibana

1，部署

# 下载安装包
[root@\ es02~]# wget https://artifacts.elastic.co/downloads/kibana/kibana-7.12.1-x86_64.rpm


# 安装
[root@\ es02~]# yum install -y kibana-7.12.1-x86_64.rpm 

# 修改配置文件
[root@\ es02~]# grep -E '^[^#]' /etc/kibana/kibana.yml 
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://172.16.1.70:9200"]


# 启动
[root@\ es02~]# systemctl start kibana.service 



# 查看是否启动
[root@\ es02~]# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:5601            0.0.0.0:*               LISTEN      13754/node          


# 访问
见图a