美文网首页
6、filebeat->kafka->logstash->es

6、filebeat->kafka->logstash->es

作者: 阿fong | 来源:发表于2020-12-10 11:48 被阅读0次

    一、部署忽略
    参考
    https://www.jianshu.com/p/a6084c9450c6

    二、开启system模块
    filebeat modules enable system

    三、配置filebeat

    filebeat.inputs:
    - type: log
      enabled: false
      paths:
        - /var/log/*.log
    
    - type: filestream
      enabled: false
      paths:
        - /var/log/*.log
    
    filebeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: false
    
    setup.template.settings:
      index.number_of_shards: 1
    
    setup.kibana:
      host: "192.168.18.13:5601"
    setup.template.overwrite: true
    setup.template.enabled: true
    setup.ilm.enabled: false
    
    output.kafka:
      hosts: ["192.168.18.15:9092","192.168.18.16:9092"]
      topic: "linux"  
    
    processors:
      - add_host_metadata:
          when.not.contains.tags: forwarded
    

    四、配置logsatsh
    1、input.conf

    input {
        kafka {
            bootstrap_servers => "192.168.18.15:9092,192.168.18.16:9092"
            topics => ["linux"]
            add_field => { type_name => "linux_system" }
            consumer_threads => 5
            codec => json               
        }
    }
    

    2、output.conf

    output {
            elasticsearch {
                hosts => ["192.168.18.13:9200"]
                index => "test-os-linux-%{+YYYY.MM.dd}"
            }
    }
    

    五、在kibana中查看索引和数据


    image.png
    image.png

    日志时间为logstash收到的时间,后面优化

    六、在kafka中查看消息的命令

    /etc/kafka/kafka_2.12-2.2.2/bin/kafka-console-consumer.sh --bootstrap-server 192.168.18.16:9092 --topic linux --from-beginning
    

    相关文章

      网友评论

          本文标题:6、filebeat->kafka->logstash->es

          本文链接:https://www.haomeiwen.com/subject/yptcgktx.html