美文网首页
Windows系统日志收集

Windows系统日志收集

作者: 洮生 | 来源:发表于2017-08-03 18:33 被阅读0次

方案

采用nxlog+fluentd+elasticsearch组件。nxlog 是用 C 语言写的一个跨平台日志收集处理软件，其内部支持使用 Perl 正则和语法来进行数据结构化和逻辑判断操作。nxlog负责收集Windows系统日志，并将日志传给fluentd，fluentd将这些日志信息抛给elasticsearch。fluentd官方文档Collecting Log Data from Windows中也是推荐使用该方案的。

预先准备

  1. Windows服务器（本次使用的是Win10）。
  2. Linux服务器（本次使用的CentOS7）。
  3. 下载elasticsearch并安装在Linux服务器上。

环境搭建步骤

Windows上安装nxlog

  1. 下载Windows版本的nxlog。
  2. 直接点击next安装即可。

补充：nxlog默认是安装在C:\Program Files (x86)\nxlog目录中。

Linux上安装fluentd

  1. 若从rpm库安装，执行如下命令：
$ curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-  agent2.sh | sh
  1. 常用命令：
$ /etc/init.d/td-agent start
$ /etc/init.d/td-agent stop
$ /etc/init.d/td-agent restart
$ /etc/init.d/td-agent status

测试

修改配置文件

  1. 修改td-agent配置文件/etc/td-agent/td-agent.conf

    $ vim /etc/td-agent/td-agent.conf

    增添如下配置并保存：

    <source>
   @type tcp
   format json
   port 5140
   tag windowslog
 </source>
 <match windowslog>
   type elasticsearch
 </match>

  2. 修改nxlog配置文件C:\Program Files (x86)\nxlog\conf
    改后的文件如下所示：

    define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension _syslog>
    Module      xm_syslog
</Extension>

<Extension json>
    Module xm_json
</Extension>
#日志输入端
<Input in>
    Module im_msvistalog 
    #For windows 2003 and earlier use the following:
    #Module im_mseventlog
    Exec to_json(); 
</Input>
#日志输出端
<Output out>
    Module om_tcp
    Host 192.168.159.133    #安装有td-agent的服务器地址
    Port 5140               #此端口要与td-agent监听端口一致
</Output> 
<Output outfile>
    Module om_file
    File   "D:\logs\log"    #为了便于观察，将收集到的系统日志存放在该位置下
</Output>
#绑定某输入到具体某输出
<Route r1>
    Path in => out
</Route>
<Route r2>
    Path in => outfile
</Route>

启动服务

  1. 启动td-agent：

    $ /etc/init.d/td-agent start

  2. 启动nxlog，使用powershell进入到C:\Program Files (x86)\nxlog目录下，执行命令：

    C:\Program Files (x86)\nxlog> .\\nxlog.exe -f -c ./conf/nxlog.conf

    部分输出如下：

    2017-07-13 14:43:30 INFO connecting to 192.168.159.133:5140
2017-07-13 14:43:30 INFO nxlog-ce-2.9.1716 started

结果

在Linux控制台查看elasticsearch中的索引：

[root@localhost ~]# curl 'localhost:9200/_cat/indices?v'
health status index               uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   fluentd             c05Ykvp_RPebnOppfsa1_w   5   1          2            0     67.7kb         67.7kb
yellow open   .kibana             Zldk9n-BSweLtaeVNMsNtg   1   1          8            0       54kb           54kb

查看fluentd索引中的数据，部分输出如下：

[root@localhost ~]# curl -XGET 'localhost:9200/fluentd/_search?pretty'
{
  "took" : 363,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 68,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "fluentd",
        "_type" : "fluentd",
        "_id" : "AV06sthgdjHN6maqugXM",
        "_score" : 1.0,
        "_source" : {
          "EventTime" : "2017-07-13 14:45:59",
          "Hostname" : "LAPTOP-AQ1VQEMH",
          "Keywords" : 576460752303423488,
          "EventType" : "INFO",
          "SeverityValue" : 2,
          "Severity" : "INFO",
          "EventID" : 20,
          "SourceName" : "Microsoft-Windows-Audio",
          "ProviderGuid" : "{AE4BD3BE-F36F-45B6-8D21-BDD6FB832853}",
          "Version" : 0,
          "Task" : 116,
          "OpcodeValue" : 0,
          "RecordNumber" : 113490,
          "ProcessID" : 6804,
          "ThreadID" : 13988,
          "Channel" : "Microsoft-Windows-Audio/PlaybackManager",
          "Domain" : "LAPTOP-AQ1VQEMH",
          "AccountName" : "WHL",
          "UserID" : "S-1-5-21-2442955922-1256182451-2774828708-1001",
          "AccountType" : "User",
          "Message" : "格式: 1\r\n采样率 : 22050Hz\r\n已卸载: false",
          "Opcode" : "信息",
          "Format" : "1",
          "SamplingRate" : "22050",
          "bAudioSrvStreamResourceType" : "false",
          "EventReceivedTime" : "2017-07-13 14:46:00",
          "SourceModuleName" : "in",
          "SourceModuleType" : "im_msvistalog"
        }
      },
      {
        "_index" : "fluentd",
        "_type" : "fluentd",
        "_id" : "AV06uizFdjHN6maqugXh",
        "_score" : 1.0,
        "_source" : {
          "EventTime" : "2017-07-13 14:53:26",
          "Hostname" : "LAPTOP-AQ1VQEMH",
          "Keywords" : 576460752303423488,
          "EventType" : "INFO",
          "SeverityValue" : 2,
          "Severity" : "INFO",
          "EventID" : 505,
          "SourceName" : "Microsoft-Windows-Application-Experience",
          "ProviderGuid" : "{EEF54E71-0661-422D-9A98-82FD4940B820}",
          "Version" : 0,
          "Task" : 0,
          "OpcodeValue" : 0,
          "RecordNumber" : 31916,
          "ProcessID" : 14276,
          "ThreadID" : 13872,
          "Channel" : "Microsoft-Windows-Application-Experience/Program-Telemetry",
          "Domain" : "LAPTOP-AQ1VQEMH",
          "AccountName" : "WHL",
          "UserID" : "S-1-5-21-2442955922-1256182451-2774828708-1001",
          "AccountType" : "User",
          "Message" : "兼容性修复程序已应用于 C:\\Users\\WHL\\AppData\\Local\\Google\\Chrome\\Application\\chrome.exe。\r\n修补程序信息: Google Chrome，{18A8B5B2-9D2F-4DB2-8307-196B5CC0CE6B}，0x80010101。",
          "Opcode" : "信息",
          "EventReceivedTime" : "2017-07-13 14:53:28",
          "SourceModuleName" : "in",
          "SourceModuleType" : "im_msvistalog"
        }
      },

打开D:\logs下的log文件，部分输出如下：

{"EventTime":"2017-07-13 14:43:51","Hostname":"LAPTOP-AQ1VQEMH","Keywords":4611686018429485056,"EventType":"INFO","SeverityValue":2,"Severity":"INFO","EventID":146,"SourceName":"Microsoft-Windows-Ntfs","ProviderGuid":"{3FF37A1C-A68D-4D6E-8C9B-F79E8B16C482}","Version":1,"Task":0,"OpcodeValue":0,"RecordNumber":11792,"ProcessID":4,"ThreadID":8584,"Channel":"Microsoft-Windows-Ntfs/Operational","Domain":"NT AUTHORITY","AccountName":"SYSTEM","UserID":"S-1-5-18","AccountType":"Well Known Group","Message":"IO latency summary:\r\n\r\n           Volume Id: {4E0E6863-60CA-4F3A-ADC1-EA02A7E953E3}\r\n           Volume name: C:\r\n           Is boot volume: true\r\n\r\n           Interval duration: 3627921184 us\r\n\r\n           Non-cached reads:\r\n                     IO count: 7387\r\n                     Total bytes: 466456052\r\n                     Avg latency: 8210412 ns\r\n\r\n           Non-cached writes: \r\n                     IO count: 24179\r\n                     Total bytes: 412844919\r\n                     Avg latency: 3188872 ns\r\n\r\n           File flushes: \r\n                     IO count: 1622\r\n                     Avg latency: 17203859 ns\r\n\r\n           Volume flushes: \r\n                     IO count: 0\r\n                     Avg latency: 0 ns\r\n\r\n           File level trims: \r\n                     IO count: 1\r\n                     Total bytes: 147374080\r\n                     Extents count: 1\r\n                     Avg latency: 1947000 ns\r\n\r\n           Volume trims: \r\n                     IO count: 324\r\n                     Total bytes: 1420537856\r\n                     Extents count: 1548\r\n                     Avg latency: 1463830 ns\r\n\r\n For more details see the details tab.\r\n","Opcode":"信息","VolumeCorrelationId":"{4E0E6863-60CA-4F3A-ADC1-EA02A7E953E3}","VolumeNameLength":"2","VolumeName":"C:","IsBootVolume":"true","HighIoLatencyCount":"0","IntervalDurationUs":"3627921184","NCReadIOCount":"7387","NCReadTotalBytes":"466456052","NCReadAvgLatencyNs":"8210412","NCWriteIOCount":"24179","NCWriteTotalBytes":"412844919","NCWriteAvgLatencyNs":"3188872","FileFlushCount":"1622","FileFlushAvgLatencyNs":"17203859","VolumeFlushCount":"0","VolumeFlushAvgLatencyNs":"0","FileLevelTrimCount":"1","FileLevelTrimTotalBytes":"147374080","FileLevelTrimExtentsCount":"1","FileLevelTrimAvgLatencyNs":"1947000","VolumeTrimCount":"324","VolumeTrimTotalBytes":"1420537856","VolumeTrimExtentsCount":"1548","VolumeTrimAvgLatencyNs":"1463830","IoBucketsCount":"48","TotalBytesBucketsCount":"44","ExtentsBucketsCount":"4","IoCount":"35","TotalLatencyUs":"51080","TotalBytes":"1208320","TrimExtentsCount":"1537","IoTypeIndex":"0","EventReceivedTime":"2017-07-13 14:43:52","SourceModuleName":"in","SourceModuleType":"im_msvistalog"}
{"EventTime":"2017-07-13 14:43:51","Hostname":"LAPTOP-AQ1VQEMH","Keywords":4611686018429485056,"EventType":"INFO","SeverityValue":2,"Severity":"INFO","EventID":146,"SourceName":"Microsoft-Windows-Ntfs","ProviderGuid":"{3FF37A1C-A68D-4D6E-8C9B-F79E8B16C482}","Version":1,"Task":0,"OpcodeValue":0,"RecordNumber":11793,"ProcessID":4,"ThreadID":8584,"Channel":"Microsoft-Windows-Ntfs/Operational","Domain":"NT AUTHORITY","AccountName":"SYSTEM","UserID":"S-1-5-18","AccountType":"Well Known Group","Message":"IO latency summary:\r\n\r\n           Volume Id: {EFA1B98E-1BD3-45F1-A956-882127C35622}\r\n           Volume name: D:\r\n           Is boot volume: false\r\n\r\n           Interval duration: 3627921030 us\r\n\r\n           Non-cached reads:\r\n                     IO count: 11248\r\n                     Total bytes: 820325912\r\n                     Avg latency: 4644197 ns\r\n\r\n           Non-cached writes: \r\n                     IO count: 7992\r\n                     Total bytes: 242775864\r\n                     Avg latency: 15979689 ns\r\n\r\n           File flushes: \r\n                     IO count: 49\r\n                     Avg latency: 10135571 ns\r\n\r\n           Volume flushes: \r\n                     IO count: 0\r\n                     Avg latency: 0 ns\r\n\r\n           File level trims: \r\n                     IO count: 0\r\n                     Total bytes: 0\r\n                     Extents count: 0\r\n                     Avg latency: 0 ns\r\n\r\n           Volume trims: \r\n                     IO count: 2\r\n                     Total bytes: 16384\r\n                     Extents count: 4\r\n                     Avg latency: 893000 ns\r\n\r\n For more details see the details tab.\r\n","Opcode":"信息","VolumeCorrelationId":"{EFA1B98E-1BD3-45F1-A956-882127C35622}","VolumeNameLength":"2","VolumeName":"D:","IsBootVolume":"false","HighIoLatencyCount":"0","IntervalDurationUs":"3627921030","NCReadIOCount":"11248","NCReadTotalBytes":"820325912","NCReadAvgLatencyNs":"4644197","NCWriteIOCount":"7992","NCWriteTotalBytes":"242775864","NCWriteAvgLatencyNs":"15979689","FileFlushCount":"49","FileFlushAvgLatencyNs":"10135571","VolumeFlushCount":"0","VolumeFlushAvgLatencyNs":"0","FileLevelTrimCount":"0","FileLevelTrimTotalBytes":"0","FileLevelTrimExtentsCount":"0","FileLevelTrimAvgLatencyNs":"0","VolumeTrimCount":"2","VolumeTrimTotalBytes":"16384","VolumeTrimExtentsCount":"4","VolumeTrimAvgLatencyNs":"893000","IoBucketsCount":"35","TotalBytesBucketsCount":"32","ExtentsBucketsCount":"1","IoCount":"2902","TotalLatencyUs":"1952578","TotalBytes":"41503232","TrimExtentsCount":"4","IoTypeIndex":"0","EventReceivedTime":"2017-07-13 14:43:52","SourceModuleName":"in","SourceModuleType":"im_msvistalog"}
{"EventTime":"2017-07-13 14:45:52","Hostname":"LAPTOP-AQ1VQEMH","Keywords":576460752303423488,"EventType":"INFO","SeverityValue":2,"Severity":"INFO","EventID":20,"SourceName":"Microsoft-Windows-Audio","ProviderGuid":"{AE4BD3BE-F36F-45B6-8D21-BDD6FB832853}","Version":0,"Task":116,"OpcodeValue":0,"RecordNumber":113487,"ProcessID":6804,"ThreadID":13716,"Channel":"Microsoft-Windows-Audio/PlaybackManager","Domain":"LAPTOP-AQ1VQEMH","AccountName":"WHL","UserID":"S-1-5-21-2442955922-1256182451-2774828708-1001","AccountType":"User","Message":"格式: 1\r\n采样率 : 22050Hz\r\n已卸载: false","Opcode":"信息","Format":"1","SamplingRate":"22050","bAudioSrvStreamResourceType":"false","EventReceivedTime":"2017-07-13 14:45:54","SourceModuleName":"in","SourceModuleType":"im_msvistalog"}
{"EventTime":"2017-07-13 14:45:59","Hostname":"LAPTOP-AQ1VQEMH","Keywords":576460752303423488,"EventType":"INFO","SeverityValue":2,"Severity":"INFO","EventID":20,"SourceName":"Microsoft-Windows-Audio","ProviderGuid":"{AE4BD3BE-F36F-45B6-8D21-BDD6FB832853}","Version":0,"Task":116,"OpcodeValue":0,"RecordNumber":113488,"ProcessID":6804,"ThreadID":12172,"Channel":"Microsoft-Windows-Audio/PlaybackManager","Domain":"LAPTOP-AQ1VQEMH","AccountName":"WHL","UserID":"S-1-5-21-2442955922-1256182451-2774828708-1001","AccountType":"User","Message":"格式: 1\r\n采样率 : 22050Hz\r\n已卸载: false","Opcode":"信息","Format":"1","SamplingRate":"22050","bAudioSrvStreamResourceType":"false","EventReceivedTime":"2017-07-13 14:46:00","SourceModuleName":"in","SourceModuleType":"im_msvistalog"}
{"EventTime":"2017-07-13 14:45:59","Hostname":"LAPTOP-AQ1VQEMH","Keywords":576460752303423488,"EventType":"INFO","SeverityValue":2,"Severity":"INFO","EventID":20,"SourceName":"Microsoft-Windows-Audio","ProviderGuid":"{AE4BD3BE-F36F-45B6-8D21-BDD6FB832853}","Version":0,"Task":116,"OpcodeValue":0,"RecordNumber":113489,"ProcessID":6804,"ThreadID":9620,"Channel":"Microsoft-Windows-Audio/PlaybackManager","Domain":"LAPTOP-AQ1VQEMH","AccountName":"WHL","UserID":"S-1-5-21-2442955922-1256182451-2774828708-1001","AccountType":"User","Message":"格式: 1\r\n采样率 : 22050Hz\r\n已卸载: false","Opcode":"信息","Format":"1","SamplingRate":"22050","bAudioSrvStreamResourceType":"false","EventReceivedTime":"2017-07-13 14:46:00","SourceModuleName":"in","SourceModuleType":"im_msvistalog"}
{"EventTime":"2017-07-13 14:45:59","Hostname":"LAPTOP-AQ1VQEMH","Keywords":576460752303423488,"EventType":"INFO","SeverityValue":2,"Severity":"INFO","EventID":20,"SourceName":"Microsoft-Windows-Audio","ProviderGuid":"{AE4BD3BE-F36F-45B6-8D21-BDD6FB832853}","Version":0,"Task":116,"OpcodeValue":0,"RecordNumber":113490,"ProcessID":6804,"ThreadID":13988,"Channel":"Microsoft-Windows-Audio/PlaybackManager","Domain":"LAPTOP-AQ1VQEMH","AccountName":"WHL","UserID":"S-1-5-21-2442955922-1256182451-2774828708-1001","AccountType":"User","Message":"格式: 1\r\n采样率 : 22050Hz\r\n已卸载: false","Opcode":"信息","Format":"1","SamplingRate":"22050","bAudioSrvStreamResourceType":"false","EventReceivedTime":"2017-07-13 14:46:00","SourceModuleName":"in","SourceModuleType":"im_msvistalog"}
{"EventTime":"2017-07-13 14:45:59","Hostname":"LAPTOP-AQ1VQEMH","Keywords":576460752303423488,"EventType":"INFO","SeverityValue":2,"Severity":"INFO","EventID":20,"SourceName":"Microsoft-Windows-Audio","ProviderGuid":"{AE4BD3BE-F36F-45B6-8D21-BDD6FB832853}","Version":0,"Task":116,"OpcodeValue":0,"RecordNumber":113491,"ProcessID":6804,"ThreadID":11340,"Channel":"Microsoft-Windows-Audio/PlaybackManager","Domain":"LAPTOP-AQ1VQEMH","AccountName":"WHL","UserID":"S-1-5-21-2442955922-1256182451-2774828708-1001","AccountType":"User","Message":"格式: 1\r\n采样率 : 22050Hz\r\n已卸载: false","Opcode":"信息","Format":"1","SamplingRate":"22050","bAudioSrvStreamResourceType":"false","EventReceivedTime":"2017-07-13 14:46:00","SourceModuleName":"in","SourceModuleType":"im_msvistalog"}

相关文章

  • elk怎么收集windows IIS日志

    elk 怎么收集windows系统日志、IIS日志，求解

  • Windows系统日志收集

    方案 采用nxlog+fluentd+elasticsearch组件。nxlog 是用 C 语言写的一个跨平台日志...

  • Windows系统日志收集

    方案 采用nxlog+fluentd+elasticsearch组件。nxlog 是用 C 语言写的一个跨平台日志...

  • Windows事件日志简要解析

    Windows系统日志 简介： Windows操作系统在运行过程中会记录大量日志信息。这些日志主要包括Window...

  • Splunk + Forwarder 收集分析Windows系统

    本文基于Splunk Forwarder自动收集Windows日志，将日志发送到Splunk做统一收集，并建立简单...

  • 查看windows开关机日志

    查看windows开关机日志 【事件查看器】-->【windows日志】-->【系统】 选择右边【筛选当前日志】 ...

  • 基于Flume的日志收集系统架构和设计

    1 日志收集系统简介日志收集是大数据的基石。 许多公司的业务平台每天都会产生大量的日志数据。收集业务日志数据，供离...

  • 5.4 痕迹清理

    5.4.1. Windows 操作日志：3389登录列表、文件访问日志、浏览器日志、系统事件 登录日志：系统安全日...

  • php错误处理之错误记录日志

    阅读原文 在一些公司里面，有专门的日志收集系统，日志收集系统会在背后默默的帮你收集错误、警告、提示。 也有些公司没...

  • 03 hadoop 日志收集

    0 目标 收集系统 docker 137个服务日志中错误日志， 根据服务名，跟日期显示 1 flume 收集日志到...

网友评论

      本文标题：Windows系统日志收集

      本文链接：https://www.haomeiwen.com/subject/pllbkxtx.html

      延伸阅读

      深度阅读

      • 您也可以注册成为美文阅读网的作者，发表您的原创作品、分享您的心情！

      栏目导航

      热点阅读

      关于我们|服务条款|联系我们|Windows系统日志收集|投稿指南|网站地图|RSS订阅|排版工具|手机版

      提供经典美文摘抄,优美散文欣赏,现代诗歌精选,短篇小说,心情随笔,表白情书范文,故事会在线阅读欣赏

      Copyright © 2014-2023 Haomeiwen.com All Rights Reserved. 好美文阅读网 版权所有

      备案信息：桂公网安备 45052102000051号 · 桂ICP备13007215号-3

      本站所收录作品、热点评论等信息部分来源互联网，目的只是为了系统归纳学习和传递资讯

      所有作品版权归原创作者所有，与本站立场无关，如不慎侵犯了你的权益，请联系我们告知，我们将做删除处理！