【详细】JAVA项目如何快速集成graylog日志系统

认识graylog

graylog是一个简单易用、功能较全面的日志管理工具，相比 ELK 组合， 优点：

• 部署维护简单
• 查询语法简单易懂(对比ES的语法…)
• 内置简单的告警
• 可以将搜索结果导出为 json
• UI 比较友好

1. graylog单机架构图

   
   

2. graylog集群架构

   
   

环境准备

• 操作系统

版本	值
centos	6.7
服务器ip地址	192.168.1.235

• 软件版本

软件	版本
jdk	1.8 +
mongodb	3.6
elasticsearch	5.6.10
graylog	2.4

• 关闭防火墙和SELinux

关闭防火墙：
service iptables stop
chkconfig iptables off

关闭SELinux：
vi /etc/selinux/config
把SELINUX 改成 disabled  
SELINUX=disabled  
重启后永久关闭

安装JDK

1. 下载
   jdk1.8下载地址
2. 解压
   解压jdk包，放到/usr/local/

cd /root/soft/
tar -zxvf jdk-8u181-linux-x64.tar.gz 
mv jdk1.8.0_181/ /usr/local/

配置环境变量

vim /etc/profile
在文件的最后面加上
export JAVA_HOME=/usr/local/jdk1.8.0_181
export JRE_HOME=/usr/local/jdk1.8.0_181/jre
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$PATH
然后保存退出

source /etc/profile  //让环境变量生效
然后执行 java -version
[root@master jdk1.8.0_181]# java -version
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
显示表示jdk配置正常

安装MongoDB

1. 配置MongoDB的yum源

编辑文件

vim /etc/yum.repos.d/mongodb-org-3.6.repo 

添加以下内容

[mongodb-org-3.6]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.6/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-3.6.asc

2. 安装
   清除安装源 yum clean all
   yum install -y mongodb-org

3. 安装注意地方

• 如果出现 “GPG key retrieval failed: [Errno 14] problem making ssl connection”
  解决方式：vim /etc/yum.repos.d/mongodb-org-3.6.repo 将gpgcheck=0

4. 编辑配置文件

# 编辑默认配置文件
vim /etc/mongod.conf

# mongod.conf

# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

# Where and how to store data.
storage:
  dbPath: /var/lib/mongo
  journal:
    enabled: true
#  engine:
#  mmapv1:
#  wiredTiger:

# how the process runs
processManagement:
  fork: true  # fork and run in background
  pidFilePath: /var/run/mongodb/mongod.pid  # location of pidfile
  timeZoneInfo: /usr/share/zoneinfo

# network interfaces
net:
  port: 27017
  bindIp: 0.0.0.0  # mongod绑定的ip地址，此处要修改


#security:

#operationProfiling:

#replication:

#sharding:

## Enterprise-Only Options

#auditLog:

5. 相关操作

# 添加服务自启动
chkconfig --add mongod

# 启动
service mongod start 

# 停止
service mongod stop 

# 重启
service mongod restart 

6. 验证mongod是否安装成功

[root@master ~]# netstat -ano| grep 27017
tcp        0      0 0.0.0.0:27017               0.0.0.0:*                   LISTEN      off (0.00/0/0)

安装Elasticsearch

1. 导入验证文件

 rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

2. 添加源文件
   vim /etc/yum.repos.d/elasticsearch.repo

[elasticsearch-5.x]
name=Elasticsearch repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

3. 安装

yum install -y elasticsearch 

4. 修改配置文件
   # vim /etc/elasticsearch/elasticsearch.yml

cluster.name: graylog
network.host: 192.168.1.235

5. 相关操作

# 添加服务自启动
chkconfig --add elasticsearch

# 启动
service elasticsearch start 

# 停止
service elasticsearch stop 

# 重启
service elasticsearch restart 

6. 检查运行服务端口

[root@master ~]# ps -ef | grep elastics
497        1419      1  0 09:13 ?        00:00:47 /usr/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid -d -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.data=/var/lib/elasticsearch -Edefault.path.conf=/etc/elasticsearch
root       1977   1676  0 11:19 pts/1    00:00:00 grep elastics
[root@master ~]# netstat -tulnp | grep 1419
tcp        0      0 ::ffff:192.168.1.235:9200   :::*                        LISTEN      1419/java           
tcp        0      0 ::ffff:192.168.1.235:9300   :::*                        LISTEN      1419/java  

7. 检查elasticsearch状态

[root@master ~]# curl -X GET http://192.168.1.235:9200
{
  "name" : "_zZYaCG",
  "cluster_name" : "graylog",
  "cluster_uuid" : "uwd67mRTQmaOuHbbypOohw",
  "version" : {
    "number" : "5.6.10",
    "build_hash" : "b727a60",
    "build_date" : "2018-06-06T15:48:34.860Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.1"
  },
  "tagline" : "You Know, for Search"
}

8. 安装注意地方

• 问题1：root用户启动的时候报错“Could not find any executable java binary. Please install java in your PATH or set JAVA_HOME”
  解决方式： ln -s /usr/java/jdk1.8.0_65/bin/java（jdk路径） /usr/bin/java 做一个软连接

• 问题2：[WARN ][o.e.b.JNANatives ] unable to install syscall filter:
  java.lang.UnsupportedOperationException: seccomp unavailable: requires kernel 3.5+ with CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER compiled in
  at org.elasticsearch.bootstrap.Seccomp.linuxImpl(Seccomp.java:349) ~[elasticsearch-5.0.0.jar:5.0.0]
  at org.elasticsearch.bootstrap.Seccomp.init(Seccomp.java:630) ~[elasticsearch-5.0.0.jar:5.0.0]
  解决方式：使用centos 7 linux版本，就不会出现此类问题了
  如果用centos6版本的，则在elasticsearch.yml中配置bootstrap.system_call_filter为false，注意要在Memory下面:
  bootstrap.memory_lock: false
  bootstrap.system_call_filter: false

• 问题3：ERROR: bootstrap checks failed
  max file descriptors [4096] for elasticsearch process likely too low, increase to at least [65536]
  max number of threads [1024] for user [lishang] likely too low, increase to at least [2048]
  解决方式：切换到root用户，编辑limits.conf 添加类似如下内容
  vi /etc/security/limits.conf
  添加如下内容:

* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096

• 问题4：max number of threads [1024] for user [lish] likely too low, increase to at least [2048]
  解决方式：切换到root用户，进入limits.d目录下修改配置文件。
  vi /etc/security/limits.d/90-nproc.conf
  修改如下内容：

* soft nproc 1024

修改为

* soft nproc 2048

• 问题5： max file descriptors [4096] for elasticsearch process likely too low, increase to at least [65536]
  解决方式：修改切换到root用户修改配置limits.conf 添加下面两行
  命令:vi /etc/security/limits.conf

*        hard    nofile           65536
*        soft    nofile           65536

• 问题6： Starting elasticsearch: Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000085330000, 2060255232, 0) failed; error='Cannot allocate memory' (errno=12)
  解决方式：

cd /etc/elasticsearch/

由于elasticsearch5.0默认分配jvm空间大小为2g，修改jvm空间分配

vim jvm.options 
-Xms2g
-Xmx2g

修改成

-Xms512m
-Xmx512m

安装graylog

1. 安装源

 rpm -Uvh https://packages.graylog2.org/repo/packages/graylog-2.4-repository_latest.rpm

2. 安装

yum install -y graylog-server

3. 生成password_secret

[root@master ~]# yum install -y pwgen 
[root@master ~]# pwgen -N 1 -s 96
77K2GVzQeM2pFV4fNgrt5e5jxhz8X3HuR5OlLwuPEAA8XJux0fkoff97GeaNlQQDWmXCiHplY6MMzwwqDXapcXeNC5qZDHs9

4. 生成root_password_sha2

[root@master ~]# echo -n 123456 | sha256sum
8d969eef6ecad3c29a3a629280e686cf0c3f5d5a86aff3ca12020c923adc6c92  -

5. 编辑配置文件
   vim /etc/graylog/server/server.conf

password_secret = 77K2GVzQeM2pFV4fNgrt5e5jxhz8X3HuR5OlLwuPEAA8XJux0fkoff97GeaNlQQDWmXCiHplY6MMzwwqDXapcXeNC5qZDHs9
root_password_sha2 = 8d969eef6ecad3c29a3a629280e686cf0c3f5d5a86aff3ca12020c923adc6c92
rest_listen_uri = http://0.0.0.0:9000/api/
web_listen_uri = http://0.0.0.0:9000/
root_timezone = Asia/Shanghai
elasticsearch_hosts = http://192.168.1.235:9200
mongodb_uri = mongodb://localhost/graylog  （因为mongod和graylog装同一台机器，这个不需要改动）

6. 相关操作

# 启动
service graylog-server start

# 停止
service graylog-server stop

# 重启
service graylog-server restart

7. 配置日志收集

• 浏览器输入：http://192.168.1.235:9000

  
  

• 输入账号：admin 密码：123456

  
  

• 点击菜单“System”--》“Inputs”

  
  

• 新建新的Input

  
  
  

8. 启动graylog遇到的坑
   当使用腾讯云那些云服务器的时候，发现bind公网ip后启动不了，不可用。如果绑定在0.0.0.0的ip上，可以启动，但是访问http://公网ip:9000的时候，出现一个错误提示：

Server currently unavailable

We are experiencing problems connecting to the Graylog server running on [http://192.168.1.1:9000/api/ 4](http://192.168.1.1:9000/api/). Please verify that the server is healthy and working correctly.

You will be automatically redirected to the previous page once we can connect to the server.

Do you need a hand? We can help you.
Less details

This is the last response we received from the server:

Error message
Bad request
Original Request
GET [http://192.168.1.1:9000/api/system/sessions 10](http://192.168.1.1:9000/api/system/sessions)
Status code
undefined
Full error message
Error: Request has been terminated Possible causes: the network is offline, Origin is not allowed by Access-Control-Allow-Origin, the page is being unloaded, etc.

这个时候，其它配置都不需要动，只需要
** vim /etc/graylog/server/server.conf**
将 web_endpoint_uri = http://公网ip:9000/api/ 上即可

graylog配置完毕，接着我们看下如何在java程序上通过logback进行日志的收集

将graylog集成在java项目上

1. 用maven新建个java的项目

   
   
   
   

2. 加入依赖

<!--logback日志-->
    <dependency>
      <groupId>ch.qos.logback</groupId>
      <artifactId>logback-classic</artifactId>
      <version>1.1.7</version>
    </dependency>

    <!--logback gelf日志收集-->
    <dependency>
      <groupId>biz.paluch.logging</groupId>
      <artifactId>logstash-gelf</artifactId>
      <version>1.11.1</version>
    </dependency>

3. 配置logback.xml

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <property resource="conf/logback.properties" />
    <property name="log.context.name" value="${log.context.name}"/>
    <property name="log.charset" value="UTF-8"/>
    <property name="log.pattern" value="[%date][%level][%thread][%class:%line] - %msg%n"/>
    <contextName>${log.context.name}</contextName>

    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder charset="${log.charset}">
            <pattern>${log.pattern}</pattern>
        </encoder>
    </appender>

    <appender name="GELF" class="biz.paluch.logging.gelf.logback.GelfLogbackAppender">
        <host>udp:192.168.1.235</host><!--graylog 服务器ip-->
        <port>11002</port><!--graylog udp端口-->
        <version>1.1</version>
        <facility>我的测试</facility>
        <extractStackTrace>true</extractStackTrace>
        <filterStackTrace>true</filterStackTrace>
        <mdcProfiling>true</mdcProfiling>
        <timestampPattern>yyyy-MM-dd HH:mm:ss,SSSS</timestampPattern>
        <maximumMessageSize>8192</maximumMessageSize>

        <!-- This are fields using MDC -->
        <mdcFields>mdcField1,mdcField2</mdcFields>
        <dynamicMdcFields>mdc.*,(mdc|MDC)fields</dynamicMdcFields>
        <includeFullMdc>true</includeFullMdc>
    </appender>

    <root>
        <level value="INFO"/>
        <appender-ref ref="STDOUT"/>
        <appender-ref ref="GELF"/>
    </root>
</configuration>

4. 加入测试代码

   
   

5. graylog上查看效果

   
   

特别是分布式应用的时候，这样查起日志来就很舒服了