Scribe安装使用

前言

Scribe是Facebook开源的日志收集工具，我建议大家还是老老实实使用ELK或者Splunk，不要瞎折腾Scribe

Centos安装Scribe真是累

更新历史

2018年09月27日 - 初稿

阅读原文 - https://wsgzao.github.io/post/scribe/

扩展阅读

Scribe - https://github.com/facebookarchive/scribe
Thrift - http://thrift.apache.org/docs/install/centos

---

Scribe简介

Scribe is a server for aggregating log data that's streamed in real
time from clients. It is designed to be scalable and reliable.

See the Scribe Wiki for documentation:
http://wiki.github.com/facebook/scribe

install scribe

OS: CentOS 7.x

This guide shows how to install the precompiled scribed binary and its dependencies on a new server. Care must be taken to use the required versions for boost (1.44.0), thrift (0.7.0) and libevent (1.4.10).

# install necessary tools
sudo yum install git make flex bison libtool automake openssl-devel libevent libevent-devel python-devel gcc-c++ byacc java-1.7.0-openjdk ant

# update autoconf
cd ~
sudo rpm -e --nodeps `rpm -qf /usr/bin/autoconf`
wget http://ftp.gnu.org/gnu/autoconf/autoconf-2.69.tar.gz
tar xzf autoconf-2.69.tar.gz
cd autoconf-2.69/
./configure
make
sudo make install
# make sure /usr/local/bin is in system PATH: add following line to ~/.bashrc
vi ~/.bashrc
export PATH=$PATH:/usr/local/bin

# install boost
cd ~
wget http://jaist.dl.sourceforge.net/project/boost/boost/1.44.0/boost_1_44_0.tar.gz
tar xzf boost_1_44_0.tar.gz
cd boost_1_44_0/
./bootstrap.sh
sudo ./bjam install

# install thrift
cd ~
git clone https://github.com/apache/thrift.git
cd thrift/
git fetch
git branch -a
git checkout 0.7.x
./bootstrap.sh
./configure
make
cp compiler/cpp/thrifty.hh compiler/cpp/thrifty.h
make
sudo make install
cd lib/py/
sudo python setup.py install

# install fb303
cd ~/thrift/contrib/fb303/
./bootstrap.sh
./configure CPPFLAGS="-DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H"
make
sudo make install
cd py/
sudo python setup.py install

# install libevent
cd ~
wget https://github.com/downloads/libevent/libevent/libevent-1.4.10-stable.tar.gz
tar xf libevent-1.4.10-stable.tar.gz 
cd libevent-1.4.10-stable
./configure
make
make install

# Copy the scribed binary and put it in the desired directory
scp scribe.tar.gz 10.65.32.134:/tmp
scp scribed 10.65.32.134:/tm
# unzip and mkdir
mkdir /opt/running
cd /opt/running
cp /tmp/scribe.tar.gz .
tar xzvf scribe.tar.gz
rm -f scribe.tar.gz
mkdir -p /data/gop/live/primary/gop
mkdir -p /data/gop/live/secondary/gop
cp /tmp/scribed /usr/local/bin/scribed
cd scribe/

[root@vm4 scribe]# cat run_scribed.sh 
#!/bin/bash

while true
do
        export LD_LIBRARY_PATH=/usr/lib:/usr/local/lib
        /usr/local/bin/scribed scribe_11315.conf 1>output/scribe.log 2>output/scribe.err
        sleep 1
done

nohup sh run_scribed.sh &

[root@sg-gop-10-65-32-134 scribe]# netstat -tlnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:10050           0.0.0.0:*               LISTEN      55310/zabbix_agentd
tcp        0      0 0.0.0.0:38422           0.0.0.0:*               LISTEN      52781/sshd
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1305/sshd
tcp6       0      0 :::10050                :::*                    LISTEN      55310/zabbix_agentd
tcp6       0      0 :::11315                :::*                    LISTEN      4294/scribed
tcp6       0      0 :::38422                :::*                    LISTEN      52781/sshd
tcp6       0      0 :::22                   :::*                    LISTEN      1305/sshd