美文网首页
vector 安装 与 日志采集

vector 安装 与 日志采集

作者: 偷油考拉 | 来源:发表于2024-03-13 15:37 被阅读0次

    一、安装

    [root@VM-201-12-centos ~]# rpm -ivh vector-0.36.0-1.x86_64.rpm 
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
        package vector-0.36.0-1.x86_64 is already installed

    [root@VM-201-12-centos ~]# rpm -ql vector
/etc/default/vector
/etc/vector/examples/docs_example.yaml
/etc/vector/examples/environment_variables.yaml
/etc/vector/examples/es_s3_hybrid.yaml
/etc/vector/examples/file_to_cloudwatch_metrics.yaml
/etc/vector/examples/file_to_prometheus.yaml
/etc/vector/examples/namespacing
/etc/vector/examples/namespacing/sinks
/etc/vector/examples/namespacing/sinks/es_cluster.yaml
/etc/vector/examples/namespacing/sinks/s3_archives.yaml
/etc/vector/examples/namespacing/sources
/etc/vector/examples/namespacing/sources/apache_logs.yaml
/etc/vector/examples/namespacing/transforms
/etc/vector/examples/namespacing/transforms/apache_parser.yaml
/etc/vector/examples/namespacing/transforms/apache_sample.yaml
/etc/vector/examples/namespacing/vector.yaml
/etc/vector/examples/prometheus_to_console.yaml
/etc/vector/examples/stdio.yaml
/etc/vector/examples/wrapped_json.yaml
/etc/vector/vector.yaml
/usr/bin/vector
/usr/lib/systemd/system/vector.service
/usr/share/doc/vector-0.36.0
/usr/share/doc/vector-0.36.0/README.md
/usr/share/licenses/vector-0.36.0
/usr/share/licenses/vector-0.36.0/LICENSE
/usr/share/vector/LICENSE-3rdparty.csv
/usr/share/vector/NOTICE
/usr/share/vector/licenses/0BSD
/usr/share/vector/licenses/Apache-2.0
/usr/share/vector/licenses/BSD-2-Clause
/usr/share/vector/licenses/BSD-3-Clause
/usr/share/vector/licenses/BSL-1.0
/usr/share/vector/licenses/CC0-1.0
/usr/share/vector/licenses/ISC
/usr/share/vector/licenses/MIT
/usr/share/vector/licenses/MPL-2.0
/usr/share/vector/licenses/OpenSSL
/usr/share/vector/licenses/Unicode-DFS-2016
/usr/share/vector/licenses/Zlib
/var/lib/vector

    二、服务管理

    1. 服务管理配置文件 /usr/lib/systemd/system/vector.service 

    [Unit]
Description=Vector
Documentation=https://vector.dev
After=network-online.target
Requires=network-online.target

[Service]
User=vector
Group=vector
ExecStartPre=/usr/bin/vector validate
ExecStart=/usr/bin/vector
ExecReload=/usr/bin/vector validate
ExecReload=/bin/kill -HUP $MAINPID
Restart=always
AmbientCapabilities=CAP_NET_BIND_SERVICE
EnvironmentFile=-/etc/default/vector
# Since systemd 229, should be in [Unit] but in order to support systemd <229,
# it is also supported to have it here.
StartLimitInterval=10
StartLimitBurst=5
[Install]
WantedBy=multi-user.target

    2. 环境变量 EnvironmentFile=-/etc/default/vector

    https://vector.dev/docs/reference/configuration/#environment-variables

    三、配置文件

    为什么使用 yaml 替换 toml 作为默认配置文件格式 - vector.dev

    --config 设置配置文件 - vector.dev，默认 /etc/vector/vector.yaml

    范例 vector.yaml：

    sources:
  catalinaout_in:
    type: file
    include:
    - /lvmdata/tomcat/tomcat-running/logs/catalina.out.*.log
    read_from: beginning
    multiline:
      start_pattern: ^[^\\s]
      condition_pattern: ^\[DEBUG]|^\[INFO\]|^\[ERROR\]|^\[WARN\]|^DEBUG|^INFO|^WARN|^ERROR|^[0-2][0-9]-\w+-2[0-1][0-9][0-9]|^2[0-1][0-9][0-9]-[0-1][0-9]-[0-2][0-9]
      mode: halt_before
      timeout_ms: 1000

transforms:
  catalinaout_tran:
    inputs:
    - catalinaout_in
    type: remap
    source: |
      msg = string!(.message)
      if starts_with(msg, "[") {
      . |= parse_regex!(.message,r'^(\[\w+\]) (?P<timestamp>\d+-\d+-\d+ \d+:\d+:\d+,\d+)')
      .timestamp, err = parse_timestamp(.timestamp, "%Y-%m-%d %H:%M:%S,%3f")
      } else if match(msg, r'^DEBUG|^INFO|^ERROR|^WARN') {
      . |= parse_regex!(.message,r'^((DEBUG|INFO|WARN|ERROR)) (?P<timestamp>\d+-\d+-\d+ \d+:\d+:\d+.\d+)')
      .timestamp, err = parse_timestamp(.timestamp, "%Y-%m-%d %H:%M:%S%.3f")
      } else if match(msg, r'^[0-9]{2}-\w{3}-[0-9]{4}') {
      . |= parse_regex!(.message,r'^(?P<timestamp>\d+-\w+-\d+ \d+:\d+:\d+.\d+)')
      .timestamp, err = parse_timestamp(.timestamp, "%d-%b-%Y %H:%M:%S%.3f")
      } else {
      . |= parse_regex!(.message,r'^(?P<timestamp>\d+-\d+-\d+ \d+:\d+:\d+.\d+)')
      .timestamp, err = parse_timestamp(.timestamp, "%Y-%m-%d %H:%M:%S%.3f")
      }

sinks:
  victorialogs:
    inputs:
    - catalinaout_tran
    healthcheck:
      enabled: false
    type: elasticsearch
    endpoints:
    - http://10.41.201.5:9428/insert/elasticsearch/
    api_version: v8
    mode: bulk
    compression: none
    request:
      headers:
        AccountID: '0'
        ProjectID: '0'
    query:
      _stream_fields: host,file
      _msg_field: message
      _time_field: timestamp
  openobserve:
    inputs:
    - catalinaout_tran
    healthcheck:
      enabled: false
    type: http
    uri: http://10.41.201.7:5080/api/default/core_rh/_json
    method: post
    auth:
      strategy: basic
      user: admin@my.com
      password: admin
    encoding:
      codec: json
      timestamp_format: rfc3339

    复杂日志在转换处理的时候，可能会报错，日志写不进去，丢了不少。transforms 还是用在特定场合吧。

    四、子目录/多配置文件

    https://vector.dev/docs/reference/configuration/#multiple-files

    在 /etc/vector 目录下，创建子目录：sources、tranforms、sinks，范例如下：

    [root@VM-1-11-centos ~]# tree /etc/vector/
/etc/vector/
|-- sinks
|   |-- catalina-openobserve.yaml
|   |-- catalina-vlog.yaml
|   `-- test-console.yaml
|-- sources
|   |-- src_catalina.yaml
|   `-- src_test.yaml.bak
|-- transforms
|   `-- trans_add_filed.yaml
`-- vector.yaml

3 directories, 7 files
    • /etc/vector.yaml，只需要包含一些全局配置：
    data_dir: /var/lib/vector/
    • sources/src_catalina.yaml
        type: file
    include:
    - /lvmdata/tomcat/tomcat-running/logs/catalina.out.*.log
    read_from: beginning
    multiline:
      start_pattern: ^[^\\s]
      condition_pattern: ^\[DEBUG]|^\[INFO\]|^\[ERROR\]|^\[WARN\]|^DEBUG|^INFO|^WARN|^ERROR|^[0-2][0-9]-\w+-2[0-1][0-9][0-9]|^2[0-1][0-9][0-9]-[0-1][0-9]-[0-2][0-9]
      mode: halt_before
      timeout_ms: 1000
    host_key: "host"
    • transforms/trans_add_filed.yaml
        inputs:
    - src_*
    type: remap
    source: |
      .hostip = "10.41.1.11"
      .service = "core_rh"

    增加 hostip 和 service field，便于检索。

    • sinks/catalina-vlog.yaml 输出到 victorialogs
        inputs:
    - trans_*
    healthcheck:
      enabled: false
    type: elasticsearch
    endpoints:
    - http://10.41.201.5:9428/insert/elasticsearch/
    api_version: v8
    mode: bulk
    compression: none
    request:
      headers:
        AccountID: '0'
        ProjectID: '0'
    query:
      _stream_fields: host,file,hostip,service
      _msg_field: message
      _time_field: timestamp
    • sinks/catalina-openobserve.yaml 输出到 openobserve
        inputs:
    - trans_*
    healthcheck:
      enabled: false
    type: http
    uri: http://10.41.201.7:5080/api/default/core_rh/_json
    method: post
    auth:
      strategy: basic
      user: admin@my.com
      password: admin
    encoding:
      codec: json
      timestamp_format: rfc3339
    • sinks/test-console.yaml 输出到 console，用于调试
        inputs:
    - trans_*
    type: console
    target: stdout
    encoding:
      codec: json

    inputs 以前面配置文件去除.yaml的文件名，作为 inputs的输入。如 catalina.yaml 以 catalina 作为 sinks 的inputs。支持 通配符。

    3. 试运行

    vector --config-dir /etc/vector

    4. 修改服务管理文件

    修改 /usr/lib/systemd/system/vector.service，增加 --config-dir /etc/vector。

    [Unit]
Description=Vector
Documentation=https://vector.dev
After=network-online.target
Requires=network-online.target

[Service]
User=vector
Group=vector
ExecStartPre=/usr/bin/vector validate --config-dir /etc/vector
ExecStart=/usr/bin/vector --config-dir /etc/vector
ExecReload=/usr/bin/vector validate --config-dir /etc/vector
ExecReload=/bin/kill -HUP $MAINPID
Restart=always
AmbientCapabilities=CAP_NET_BIND_SERVICE
EnvironmentFile=-/etc/default/vector
# Since systemd 229, should be in [Unit] but in order to support systemd <229,
# it is also supported to have it here.
StartLimitInterval=10
StartLimitBurst=5
[Install]
WantedBy=multi-user.target

    systemctl daemon-reload
systemctl restart vector

    相关文章

      网友评论

          本文标题：vector 安装 与 日志采集

          本文链接：https://www.haomeiwen.com/subject/hrukzdtx.html

          延伸阅读

          深度阅读

          • 您也可以注册成为美文阅读网的作者，发表您的原创作品、分享您的心情！

          栏目导航

          热点阅读

          关于我们|服务条款|联系我们|vector 安装 与 日志采集|投稿指南|网站地图|RSS订阅|排版工具|手机版

          提供经典美文摘抄,优美散文欣赏,现代诗歌精选,短篇小说,心情随笔,表白情书范文,故事会在线阅读欣赏

          Copyright © 2014-2023 Haomeiwen.com All Rights Reserved. 好美文阅读网 版权所有

          备案信息：桂公网安备 45052102000051号 · 桂ICP备13007215号-3

          本站所收录作品、热点评论等信息部分来源互联网，目的只是为了系统归纳学习和传递资讯

          所有作品版权归原创作者所有，与本站立场无关，如不慎侵犯了你的权益，请联系我们告知，我们将做删除处理！