ELK

作者: XiaoMing丶 | 来源:发表于2019-03-21 02:58 被阅读0次

需求背景
业务发展越来越庞大，服务器越来越多
各种访问日志、应用日志、错误日志量越来越多
开发人员排查问题，需要到服务器上查日志，不方便
运营人员需要一些数据，需要我们运维到服务器上分析日志

ELK介绍
官网https://www.elastic.co/cn/
中文指南https://www.gitbook.com/book/chenryn/elk-stack-guide-cn/details
ELK Stack （5.0版本之后→ Elastic Stack ==（ELK Stack + Beats）
ELK Stack包含：ElasticSearch、Logstash、Kibana
ElasticSearch是一个搜索引擎，用来搜索、分析、存储日志。它是分布式的，也就是说可以横向扩容，可以自动发现，索引自动分片，总之很强大。文档https://www.elastic.co/guide/cn/elasticsearch/guide/current/index.html
Logstash用来采集日志，把日志解析为json格式交给ElasticSearch。
Kibana是一个数据可视化组件，把处理后的结果通过web界面展示
Beats在这里是一个轻量级日志采集器，其实Beats家族有5个成员
早期的ELK架构中使用Logstash收集、解析日志，但是Logstash对内存、cpu、io等资源消耗比较高。相比 Logstash，Beats所占系统的CPU和内存几乎可以忽略不计
x-pack对Elastic Stack提供了安全、警报、监控、报表、图表于一身的扩展包，是收费的

ELK架构

二、ELK安装准备工作

准备3台机器130,132,128
角色划分：
3台全部安装elasticsearch(后续简称es) ，1主节点130，2数据节点132,128
es主130上安装kibana
1台es数据节点132上安装logstash 3台机器全部安装jdk8（openjdk即可）
yum install -y java-1.8.0-openjdk

#三个机器的hosts都改一下，方便管理
[root@minglinux-01 ~] vim /etc/hosts
···
  8 192.168.162.130 minglinux-01
  9 192.168.162.132 minglinux-02
 10 192.168.162.128 minglinux-03

#安装jdk
[root@minglinux-01 ~] java -version
java version "1.8.0_191"
Java(TM) SE Runtime Environment (build 1.8.0_191-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode)
[root@minglinux-01 ~] which java
/usr/local/jdk1.8/bin/java

[root@minglinux-02 ~] java -version
java version "1.8.0_191"
Java(TM) SE Runtime Environment (build 1.8.0_191-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode)
[root@minglinux-02 ~] which java
/usr/local/jdk1.8/bin/java

[root@minglinux-03 ~] java -version
-bash: java: 未找到命令
[root@minglinux-03 ~] yum install -y java-1.8.0-openjdk
[root@minglinux-03 ~] java -version
openjdk version "1.8.0_201"
OpenJDK Runtime Environment (build 1.8.0_201-b09)
OpenJDK 64-Bit Server VM (build 25.201-b09, mixed mode)
[root@minglinux-03 ~] which java
/usr/bin/java

三、安装es

官方文档 https://www.elastic.co/guide/en/elastic-stack/current/installing-elastic-stack.html
以下操作3台机器上都要执行
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
vim /etc/yum.repos.d/elastic.repo //加入如下内容
[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
yum install -y elasticsearch //也可以直接下载rpm文件，然后安装
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.0.0.rpm
rpm -ivh elasticsearch-6.0.0.rpm

#3台机器配置yum源，然后yum安装elasticsearch
[root@minglinux-01 ~] rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
[root@minglinux-01 ~] vim /etc/yum.repos.d/elastic.repo
#加入以下内容
  1 [elasticsearch-6.x]
  2 name=Elasticsearch repository for 6.x packages
  3 baseurl=https://artifacts.elastic.co/packages/6.x/yum
  4 gpgcheck=1
  5 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
  6 enabled=1
  7 autorefresh=1
  8 type=rpm-md
[root@minglinux-01 ~] yum list |grep elastic
apm-server.i686                         6.6.2-1                        elasticsearch-6.x
apm-server.x86_64                       6.6.2-1                        elasticsearch-6.x
auditbeat.i686                          6.6.2-1                        elasticsearch-6.x
auditbeat.x86_64                        6.6.2-1                        elasticsearch-6.x
elastic-curator.noarch                  3.2.3-1.el7                    epel     
elasticdump.noarch                      2.2.0-2.el7                    epel     
elasticsearch.noarch                    6.6.2-1                        elasticsearch-6.x
filebeat.i686                           6.6.2-1                        elasticsearch-6.x
filebeat.x86_64                         6.6.2-1                        elasticsearch-6.x
heartbeat-elastic.i686                  6.6.2-1                        elasticsearch-6.x
heartbeat-elastic.x86_64                6.6.2-1                        elasticsearch-6.x
journalbeat.i686                        6.6.2-1                        elasticsearch-6.x
journalbeat.x86_64                      6.6.2-1                        elasticsearch-6.x
kibana.x86_64                           6.6.2-1                        elasticsearch-6.x
kibana-oss.x86_64                       6.3.0-1                        elasticsearch-6.x
logstash.noarch                         1:6.6.2-1                      elasticsearch-6.x
metricbeat.i686                         6.6.2-1                        elasticsearch-6.x
metricbeat.x86_64                       6.6.2-1                        elasticsearch-6.x
packetbeat.i686                         6.6.2-1                        elasticsearch-6.x
packetbeat.x86_64                       6.6.2-1                        elasticsearch-6.x
pcp-pmda-elasticsearch.x86_64           4.1.0-5.el7_6                  updates  
python-elasticsearch.noarch             1.9.0-1.el7                    epel     
rsyslog-elasticsearch.x86_64            8.24.0-34.el7                  base     
[root@minglinux-01 ~] yum install -y elasticsearch
[root@minglinux-01 ~] ln -s /usr/local/jdk1.8/bin/java /usr/bin/java
#报错了，做一个软连接后安装成功
could not find java; set JAVA_HOME or ensure java is in PATH
error: %pre(elasticsearch-0:6.6.2-1.noarch) scriptlet failed, exit status 1
Error in PREIN scriptlet in rpm package elasticsearch-6.6.2-1.noarch
[root@minglinux-01 ~] yum install -y elasticsearch

[root@minglinux-02 ~] rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
[root@minglinux-02 ~] vim /etc/yum.repos.d/elastic.repo
[root@minglinux-02 ~] yum install -y elasticsearch
[root@minglinux-02 ~] ln -s /usr/local/jdk1.8/bin/java /usr/bin/java
[root@minglinux-02 ~] yum install -y elasticsearch

[root@minglinux-03 ~] rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
[root@minglinux-03 ~] vim /etc/yum.repos.d/elastic.repo
[root@minglinux-03 ~] yum install -y elasticsearch

四、配置es

elasticsearch配置文件/etc/elasticsearch和/etc/sysconfig/elasticsearch
参考https://www.elastic.co/guide/en/elasticsearch/reference/6.0/rpm.html
在130上编辑配置文件vim /etc/elasticsearch/elasticsearch.yml//增加或更改
cluster.name: minglinux
node,name: minglinux-01
node.master: true//意思是该节点为主节点
node.data: false //是不是数据节点
network.host: 192.168.162.130 //0.0.0.0是监听全部ip，不安全
discovery.zen.ping.unicast.hosts: ["192.168.162.130", "192.168.162.132", "192.168.162.128"]
在132和128上同样编辑配置文件vim /etc/elasticsearch/elasticsearch.yml//增加或更改
cluster.name: minglinux
node.name: minglinux-02 或minglinux-03
node.master: false
node.data: true //是数据节点
network.host: 192.168.162.132 或192.168.162.128
discovery.zen.ping.unicast.hosts: ["192.168.162.130", "192.168.162.132", "192.168.162.128"]

#130编辑配置文件
[root@minglinux-01 ~] vim /etc/elasticsearch/elasticsearch.yml
#再对应区域添加如下内容
 18 cluster.name: minglinux
 26 node.name: minglinux-01
 30 node.master: true
 31 node.data: false
 60 network.host: 192.168.162.130
 74 discovery.zen.ping.unicast.hosts: ["192.168.162.130", "192.168.162.132", "192.168.162.128"]

#将01机器修改好的配置文件传到02和03机器，然后修改
[root@minglinux-01 ~] scp /etc/elasticsearch/elasticsearch.yml minglinux-02:/tmp/
elasticsearch.yml                                                      100% 3076     1.6MB/s   00:00    
[root@minglinux-01 ~] scp /etc/elasticsearch/elasticsearch.yml minglinux-03:/tmp/
elasticsearch.yml                                                      100% 3076     2.6MB/s   00:00    

[root@minglinux-02 ~] cp /tmp/elasticsearch
elasticsearch-6356716019382047520/ elasticsearch.yml                  
[root@minglinux-02 ~] cp /tmp/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml 
cp：是否覆盖"/etc/elasticsearch/elasticsearch.yml"？ y
[root@minglinux-02 ~] vim /etc/elasticsearch/elasticsearch.yml
···
cluster.name: minglinux
node.name: minglinux-02 
node.master: false
node.data: true
network.host: 192.168.162.132
discovery.zen.ping.unicast.hosts: ["192.168.162.130", "192.168.162.132", "192.168.162.128"]

[root@minglinux-03 ~] cp /tmp/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml
cp：是否覆盖"/etc/elasticsearch/elasticsearch.yml"？ y
[root@minglinux-03 ~] vim !$
vim /etc/elasticsearch/elasticsearch.yml
···
cluster.name: minglinux
node.name: minglinux-03
node.master: false
node.data: true
network.host: 192.168.162.128
discovery.zen.ping.unicast.hosts: ["192.168.162.130", "192.168.162.132", "192.168.162.128"]

ELK安装 – 安装x-pack（可省略）

3台机器上都要执行
cd /usr/share/elasticsearch/bin/ （可省略）
./elasticsearch-plugin install x-pack //如果速度慢，就下载x-pack压缩包（可省略）
cd /tmp/; wget https://artifacts.elastic.co/downloads/packs/x-pack/x-pack-6.0.0.zip （可省略）
./elasticsearch-plugin install file:///tmp/x-pack-6.0.0.zip （可省略）
启动elasticsearch服务
启动失败查看日志/var/log/elasticsearch/minglinux.log
systemctl enable elasticsearch.service
systemctl start elasticsearch.service
以下操作只需要在130上执行
安装x-pack后就可以为内置用户设置密码了，如下
/usr/share/elasticsearch/bin/x-pack/setup-passwords interactive （可省略）
curl localhost:9200 -u elastic //输入密码，可以查看到输出信息（可省略）

#由于x-pack是收费的就不安装了，接下来启动服务
[root@minglinux-01 ~] systemctl start elasticsearch
[root@minglinux-01 ~] ps aux |grep elasticsearch
elastic+  35666 34.5 52.4 3444748 977676 ?      Ssl  20:25   0:22 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch-4079948052346171381 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/var/log/elasticsearch/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=default -Des.distribution.type=rpm -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet
root      35840  0.0  0.0 112720   984 pts/0    R+   20:26   0:00 grep --color=auto elasticsearch
#01机器监听的端口一直看不到，而且启动起来后会自动退出，检查到有iptables规则，所以清空规则，然后在杀掉一些进程，服务器负载太高了。启动都要等好一会才看到9200和9300端口。
[root@minglinux-01 ~] systemctl stop iptables.service 
[root@minglinux-01 ~] iptables -nvL
[root@minglinux-01 ~] pkill mongod
[root@minglinux-01 ~] ps aux |grep mongo
root      41950  0.0  0.0 112720   984 pts/0    S+   21:15   0:00 grep --color=auto mongo
      

[root@minglinux-02 ~] systemctl start elasticsearch
[root@minglinux-02 ~] ps aux |grep elasticsearch
elastic+   5101 37.8 60.5 1591216 696520 ?      Ssl  20:48   0:02 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch-8969078006330607850 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/var/log/elasticsearch/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=default -Des.distribution.type=rpm -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet
root       5155 23.0  0.0 112720   984 pts/1    S+   20:48   0:00 grep --color=auto elasticsearch

[root@minglinux-03 ~] systemctl start elasticsearch
[root@minglinux-03 ~] ps aux |grep elasticsearch
elastic+  21046 69.0 74.7 1550420 746264 ?      Ssl  20:49   0:02 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch-7191065286840389837 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/var/log/elasticsearch/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=default -Des.distribution.type=rpm -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet
root      21093 36.0  0.0 112720   984 pts/0    S+   20:49   0:00 grep --color=auto elasticsearch

五、curl查看es

130上执行
curl '192.168.162.130:9200/_cluster/health?pretty' 健康检查
curl '192.168.162.130/_cluster/state?pretty' 集群详细信息
参考 http://zhaoyanblog.com/archives/732.html

[root@minglinux-01 ~] curl '192.168.162.130:9200/_cluster/health?pretty' 
{
  "cluster_name" : "minglinux",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

[root@minglinux-01 ~] curl '192.168.162.130:9200/_cluster/state?pretty' 
#太长了不贴出来了

六、安装kibanan

以下在130上执行
前面已经配置过yum源，这里就不用再配置了
yum install -y kibana
若速度太慢，可以直接下载rpm包
wget https://artifacts.elastic.co/downloads/kibana/kibana-6.0.0-x86_64.rpm
rpm -ivh kibana-6.0.0-x86_64.rpm
kibana同样也需要安装x-pack（可省略）
安装方法同elasticsearch的x-pack
cd /usr/share/kibana/bin （可省略）
./kibana-plugin install x-pack //如果这样安装比较慢，也可以下载zip文件（可省略）
wget https://artifacts.elastic.co/downloads/packs/x-pack/x-pack-6.0.0.zip//这个文件和前面下载的那个其实是一个（可省略）
./kibana-plugin install file:///tmp/x-pack-6.0.0.zip （可省略）

以下也是在130上执行
vim /etc/kibana/kibana.yml //增加
server.host: 192.168.162.130 //未安装x-pack无需密码登录，安全起见只监听内网
elasticsearch.url: "http://192.168.162.130:9200"
logging.dest: /var/log/kibana.log
touch /var/log/kibana.log; chmod 777 /var/log/kibana.log
systemctl restart kibana
浏览器里访问http://192.168.162.130:5601/
用户名elastic，密码为之前你设置过的密码（如果未安装x-pack，不需要用户名密码）
若无法输入用户名密码，查日志/var/log/kibana.log
出现错误 Status changed from uninitialized to red - Elasticsearch is still initializing the kibana index.
解决办法：curl -XDELETE http://192.168.162.130:9200/.kibana -uelastic

#安装
[root@minglinux-01 ~] yum install -y kibana

#修改配置文件
[root@minglinux-01 ~] vim /etc/kibana/kibana.yml 
#新增如下内容
  2 server.port: 5601
  8 server.host: 192.168.162.130
 28 elasticsearch.hosts: ["http://192.168.162.130:9200"]
 97 logging.dest: /var/log/kibana.log

#创建日志目录并修改权限
[root@minglinux-01 ~] touch /var/log/kibana.log; chmod 777 /var/log/kibana.log

#启动kibana
[root@minglinux-01 ~] systemctl restart kibana
[root@minglinux-01 ~] ps aux |grep kibana
kibana    49103  109  8.0 1236344 149788 ?      Rsl  22:13   0:14 /usr/share/kibana/bin/../node/bin/node --no-warnings --max-http-header-size=65536 /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml
root      49142  0.0  0.0 112720   984 pts/0    S+   22:13   0:00 grep --color=auto kibana
[root@minglinux-01 ~] netstat -lntp |grep 5601
tcp        0      0 192.168.162.130:5601    0.0.0.0:*               LISTEN      49103/node

浏览器访问

七、安装logstash

以下在132上执行
logstash目前不支持java9
直接yum安装（配置源同前面es的源）
yum install -y logstash //如果慢，就下载rpm包
wget https://artifacts.elastic.co/downloads/logstash/logstash-6.0.0.rpm
cd /usr/share/logstash/bin/（可省略）
./logstash-plugin install file:///tmp/x-pack-6.0.0.zip （可省略）

#02机器上安装logstash
[root@minglinux-02 ~] yum install -y logstash

先进行logstash收集syslog日志的一个测试

以下在132上操作
编辑配置文件 vim /etc/logstash/conf.d/syslog.conf//加入如下内容
input {
syslog {
type => "system-syslog" #这里将syslog输出到logstash监听端口10514
port => 10514 #也可以直接指定文件
}
}
output {
stdout {
codec => rubydebug
}
}
检测配置文件是否有错
cd /usr/share/logstash/bin
./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf --config.test_and_exit

以下在132上操作
前台形式启动logstash
./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf//这样可以在屏幕上查看到日志输出，不能敲命令
再开一个终端
检测是否开启10514端口：netstat -lnp |grep 10514
vi /etc/rsyslog.conf//在#### RULES下面增加一行
. @@127.0.0.1:10514
systemctl restart rsyslog
从130ssh到132上，可以在logstash前台的终端上看到ssh登录的相关日志
结束logstash，在前台的那个终端上按ctrl c

[root@minglinux-02 ~] vim /etc/logstash/conf.d/syslog.conf
#新增如下内容
input {
syslog {
type => "system-syslog"
port => 10514
}
}
output {
stdout {
codec => rubydebug
}
}

#检测配置文件是否有错
[root@minglinux-02 ~] cd /usr/share/logstash/bin
[root@minglinux-02 /usr/share/logstash/bin] ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf --config.test_and_exit
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2019-03-20T23:28:03,361][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/var/lib/logstash/queue"}
[2019-03-20T23:28:03,398][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/var/lib/logstash/dead_letter_queue"}
[2019-03-20T23:28:05,583][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[2019-03-20T23:28:20,681][INFO ][logstash.runner          ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

#前台形式启动logstash
[root@minglinux-02 /usr/share/logstash/bin] ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf
#等待在屏幕上查看日志输出，不能敲命令

#另开一个终端执行一些操作，然后查看日志是否输出到10514端口，即显示在启动logstash的终端的前台
[root@minglinux-02 /usr/share/logstash/bin] vim /etc/rsyslog.conf
#RULES下面增加一行
#### RULES ####
*.* @@192.168.162.132:10514

[root@minglinux-02 ~] systemctl restart rsyslog
[root@minglinux-02 ~] netstat -lnp |grep 10514
tcp6       0      0 :::10514                :::*                    LISTEN      5681/java           
udp        0      0 0.0.0.0:10514           0.0.0.0:*                           5681/java           

#进行ssh登录
[root@minglinux-01 ~] ssh minglinux-02
Last login: Wed Mar 20 23:33:54 2019 from 192.168.162.1
[root@minglinux-02 ~] 登出
Connection to minglinux-02 closed.
[root@minglinux-01 ~] ssh minglinux-02
Last login: Wed Mar 20 23:40:30 2019 from gitlab.example.com
[root@minglinux-02 ~] 登出
Connection to minglinux-02 closed.


#可以查看到ssh登录的相关日志
[root@minglinux-02 /usr/share/logstash/bin] ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2019-03-20T23:49:07,728][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-03-20T23:49:07,772][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.6.2"}
[2019-03-20T23:49:20,782][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-03-20T23:49:21,498][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x9b299a7 run>"}
[2019-03-20T23:49:21,719][INFO ][logstash.inputs.syslog   ] Starting syslog tcp listener {:address=>"0.0.0.0:10514"}
[2019-03-20T23:49:21,728][INFO ][logstash.inputs.syslog   ] Starting syslog udp listener {:address=>"0.0.0.0:10514"}
[2019-03-20T23:49:21,749][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-03-20T23:49:22,257][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2019-03-20T23:50:01,797][INFO ][logstash.inputs.syslog   ] new connection {:client=>"192.168.162.132:53582"}
{
           "message" => "Started Session 38 of user root.\n",
          "@version" => "1",
          "priority" => 30,
         "timestamp" => "Mar 20 23:50:01",
          "severity" => 6,
              "host" => "192.168.162.132",
              "type" => "system-syslog",
    "facility_label" => "system",
    "severity_label" => "Informational",
        "@timestamp" => 2019-03-20T15:50:01.000Z,
           "program" => "systemd",
         "logsource" => "minglinux-02",
          "facility" => 3
}
{
           "message" => "action 'action 0' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]\n",
          "@version" => "1",
          "priority" => 46,
         "timestamp" => "Mar 20 23:50:01",
          "severity" => 6,
              "host" => "192.168.162.132",
              "type" => "system-syslog",
    "facility_label" => "syslogd",
    "severity_label" => "Informational",
        "@timestamp" => 2019-03-20T15:50:01.000Z,
           "program" => "rsyslogd",
         "logsource" => "minglinux-02",
          "facility" => 5
}

八、配置logstash

以下在132上操作
后台形式启动logstash
编辑配置文件 vim /etc/logstash/conf.d/syslog.conf//配置文件内容改为如下
input {
syslog {
type => "system-syslog"
port => 10514
}
}
output {
elasticsearch {
hosts => ["192.168.162.132:9200"]
index => "system-syslog-%{+YYYY.MM}"
}
}
systemctl start logstash //启动需要一些时间，启动完成后，可以看到9600端口和10514端口已被监听
130上执行curl 'localhost:9200/_cat/indices?v' 可以获取索引信息
curl -XGET 'localhost:9200/indexname?pretty' 可以获指定索引详细信息
curl -XDELETE 'localhost:9200/logstash-xxx-*' 可以删除指定索引
浏览器访问192.168.162.130:5601，到kibana配置索引
左侧点击“Managerment”-> “Index Patterns”-> “Create Index Pattern”
Index pattern这里需要根据前面curl查询到的索引名字来写，否则下面的按钮是无法点击的

[root@minglinux-02 /usr/share/logstash/bin] vim /etc/logstash/conf.d/syslog.conf
#配置文件改为如下
input {
  syslog {
    type => "system-syslog"
    port => 10514
  } 
} 
output {
  elasticsearch {
    hosts => ["192.168.162.132:9200"]
    index => "system-syslog-%{+YYYY.MM}"
  }
}

#检测一下配置文件有没有错
[root@minglinux-02 /usr/share/logstash/bin] ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf --config.test_and_exit
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2019-03-21T00:07:23,462][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[2019-03-21T00:07:33,481][INFO ][logstash.runner          ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

#后台以服务方式启动logstash
[root@minglinux-02 /usr/share/logstash/bin] systemctl start logstash 
Failed to start logstash.service: Unit not found.
[root@minglinux-02 /usr/share/logstash/bin] /usr/share/logstash/bin/system-install /etc/logstash/startup.options systemd  #生成logstash.service
Using provided startup.options file: /etc/logstash/startup.options
Manually creating startup for specified platform: systemd
Successfully created system startup script for Logstash
[root@minglinux-02 /usr/share/logstash/bin] systemctl start logstash 
[root@minglinux-02 /usr/share/logstash/bin] ps aux |grep logstash
logstash   6481  190 33.7 3477180 387680 ?      SNsl 00:14   0:47 /bin/java -Xms1g -Xmx1g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -cp /usr/share/logstash/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/usr/share/logstash/logstash-core/lib/jars/commons-codec-1.11.jar:/usr/share/logstash/logstash-core/lib/jars/commons-compiler-3.0.8.jar:/usr/share/logstash/logstash-core/lib/jars/error_prone_annotations-2.0.18.jar:/usr/share/logstash/logstash-core/lib/jars/google-java-format-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/gradle-license-report-0.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/guava-22.0.jar:/usr/share/logstash/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-annotations-2.9.5.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-core-2.9.5.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-databind-2.9.5.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.5.jar:/usr/share/logstash/logstash-core/lib/jars/janino-3.0.8.jar:/usr/share/logstash/logstash-core/lib/jars/javassist-3.22.0-GA.jar:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.1.13.0.jar:/usr/share/logstash/logstash-core/lib/jars/jsr305-1.3.9.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-api-2.9.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.9.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-slf4j-impl-2.9.1.jar:/usr/share/logstash/logstash-core/lib/jars/logstash-core.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.commands-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.contenttype-3.4.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.expressions-3.4.300.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.filesystem-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.jobs-3.5.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.resources-3.7.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.runtime-3.7.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.app-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.common-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.preferences-3.4.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.registry-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.jdt.core-3.10.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.osgi-3.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.text-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/slf4j-api-1.7.25.jar org.logstash.Logstash --path.settings /etc/logstash
root       6509  0.0  0.0 112720   980 pts/1    R+   00:14   0:00 grep --color=auto logstash

#启动过程较久，可以看看日志是否有错
[root@minglinux-02 /usr/share/logstash/bin] less /var/log/logstash/logstash-plain.log 
#日志没有更新，判断是权限问题，更改所属用户
[root@minglinux-02 /usr/share/logstash/bin] chown logstash /var/log/logstash/logstash-plain.log
[root@minglinux-02 /usr/share/logstash/bin] chown -R logstash /var/lib/logstash
[root@minglinux-02 /usr/share/logstash/bin] systemctl restart logstash 
#终于成功了

#等到10514和9600端口被监听才是启动成功
[root@minglinux-02 /usr/share/logstash/bin] netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      4232/master         
tcp        0      0 0.0.0.0:10050           0.0.0.0:*               LISTEN      3818/zabbix_agentd  
tcp        0      0 192.168.162.132:27017   0.0.0.0:*               LISTEN      3981/mongod         
tcp        0      0 127.0.0.1:27017         0.0.0.0:*               LISTEN      3981/mongod         
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1/systemd           
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      3804/sshd           
tcp6       0      0 ::1:25                  :::*                    LISTEN      4232/master         
tcp6       0      0 127.0.0.1:9600          :::*                    LISTEN      7323/java           
tcp6       0      0 :::10050                :::*                    LISTEN      3818/zabbix_agentd  
tcp6       0      0 :::3306                 :::*                    LISTEN      4276/mysqld         
tcp6       0      0 :::111                  :::*                    LISTEN      1/systemd           
tcp6       0      0 192.168.162.132:9200    :::*                    LISTEN      5101/java           
tcp6       0      0 :::10514                :::*                    LISTEN      7323/java           
tcp6       0      0 192.168.162.132:9300    :::*                    LISTEN      5101/java           
tcp6       0      0 :::22                   :::*                    LISTEN      3804/sshd

九、kibanan上查看日志

130上执行curl '192.168.162.130:9200/_cat/indices?v' 可以获取索引信息
curl -XGET '192.168.162.130:9200/indexname?pretty' 可以获指定索引详细信息
curl -XDELETE '192.168.162.130:9200/logstash-xxx-*' 可以删除指定索引
浏览器访问192.168.162.130:5601，到kibana配置索引
左侧点击“Managerment”-> “Index Patterns”-> “Create Index Pattern”
Index pattern这里需要根据前面curl查询到的索引名字来写，否则下面的按钮是无法点击的

#获取索引信息
[root@minglinux-01 ~] curl '192.168.162.130:9200/_cat/indices?v' 
health status index                 uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .kibana_1             IpJkISvGSBecrHJR5dNMtA   1   1          3            0       24kb           12kb
green  open   system-syslog-2019.03 cWR1XimDQE6PkexAIHCYkA   5   1          5            0     81.5kb         40.7kb
[root@minglinux-01 ~]  curl -XGET '192.168.162.130:9200/system-syslog-2019.03?pretty'

到kibana配置索引查看日志

[root@minglinux-02 /usr/share/logstash/bin] tail -f /var/log/messages
Mar 21 00:38:33 minglinux-02 logstash: [2019-03-21T00:38:33,852][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x1354ed0d run>"}
Mar 21 00:38:34 minglinux-02 logstash: [2019-03-21T00:38:34,787][INFO ][logstash.inputs.syslog   ] Starting syslog udp listener {:address=>"0.0.0.0:10514"}
Mar 21 00:38:34 minglinux-02 logstash: [2019-03-21T00:38:34,792][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
Mar 21 00:38:34 minglinux-02 logstash: [2019-03-21T00:38:34,789][INFO ][logstash.inputs.syslog   ] Starting syslog tcp listener {:address=>"0.0.0.0:10514"}
Mar 21 00:38:35 minglinux-02 logstash: [2019-03-21T00:38:35,908][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
Mar 21 00:40:01 minglinux-02 systemd: Started Session 45 of user root.
Mar 21 00:40:01 minglinux-02 rsyslogd: action 'action 0' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
Mar 21 00:40:01 minglinux-02 rsyslogd: action 'action 0' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
Mar 21 00:40:01 minglinux-02 logstash: [2019-03-21T00:40:01,528][INFO ][logstash.inputs.syslog   ] new connection {:client=>"192.168.162.132:34460"}
Mar 21 00:50:01 minglinux-02 systemd: Started Session 46 of user root.

十、收集nginx日志

132上编辑配置文件 vi /etc/logstash/conf.d/nginx.conf//加入如下内容
input {
file {
path => "/tmp/elk_access.log"
start_position => "beginning"
type => "nginx"
}
}
filter {
grok {
match => { "message" => "%{IPORHOST:http_host} %{IPORHOST:clientip} - %{USERNAME:remote_user} [%{HTTPDATE:timestamp}] "(?:%{WORD:http_verb} %{NOTSPACE:http_request}(?: HTTP/%{NUMBER:http_version})?|%{DATA:raw_http_request})" %{NUMBER:response} (?:%{NUMBER:bytes_read}|-) %{QS:referrer} %{QS:agent} %{QS:xforwardedfor} %{NUMBER:request_time:float}"}
}
geoip {
source => "clientip"
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
hosts => ["192.168.162.132:9200"]
index => "nginx-test-%{+YYYY.MM.dd}"
}
}
检测配置文件是否有错
cd /usr/share/logstash/bin
./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/nginx.conf --config.test_and_exit
yum install -y nginx
vim /usr/local/nginx/conf/vhost/elk.conf//创建虚拟主机配置文件写入如下内容
server {
listen 80;
server_name elk.ming.com;
location / {
proxy_pass http://192.168.162.130:5601;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
access_log /tmp/elk_access.log main2;
}

vim /etc/nginx/nginx.conf//增加如下内容
log_format main2 '$http_host $remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$upstream_addr" $request_time';
/usr/local/nginx/sbin/nginx -t
systemctl start nginx
windows绑定hosts 192.168.162.132 elk.ming.com
浏览器访问，检查是否有日志产生
systemctl restart logstash

[root@minglinux-02 /usr/share/logstash/bin] vi /etc/logstash/conf.d/nginx.conf
#加入以下内容
input {
  file {
    path => "/tmp/elk_access.log"
    start_position => "beginning"
    type => "nginx"
  }
}
filter {
    grok {
    }
    geoip {
input {
  file {
    path => "/tmp/elk_access.log"
    start_position => "beginning"
    type => "nginx"
  }
}
filter {
    grok {
        match => { "message" => "%{IPORHOST:http_host} %{IPORHOST:clientip} - %{USERNAME:remote_user} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:http_verb} %{NOTSPACE:http_request}(?: HTTP/%{NUMBER:http_version})?|%{DATA:raw_http_request})\" %{NUMBER:response} (?:%{NUMBER:bytes_read}|-) %{QS:referrer} %{QS:agent} %{QS:xforwardedfor} %{NUMBER:request_time:float}"}
    }
    geoip {
        source => "clientip"
    }
}
output {
    stdout { codec => rubydebug }
    elasticsearch {
        hosts => ["192.168.162.132:9200"]
        index => "nginx-test-%{+YYYY.MM.dd}"
  }
}
#检测
[root@minglinux-02 /usr/share/logstash/bin] ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/nginx.conf --config.test_and_exit
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2019-03-21T02:03:17,049][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[2019-03-21T02:03:33,526][INFO ][logstash.runner          ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
#重启logstash 
[root@minglinux-02 /usr/share/logstash/bin] systemctl restart logstash 


#新建虚拟主机配置文件
[root@minglinux-02 /usr/local/nginx/conf/vhost] vim elk.conf
#内容如下
server {
            listen 80;
            server_name elk.ming.com;

            location / {
                proxy_pass      http://192.168.162.130:5601;
                proxy_set_header Host   $host;
                proxy_set_header X-Real-IP      $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            }
            access_log  /tmp/elk_access.log main2;
        }

[root@minglinux-02 /usr/local/nginx/conf/vhost] vim /usr/local/nginx/conf/nginx.conf
#加入以下内容
···
    log_format main2 '$http_host $remote_addr - $remote_user [$time_local] "$request"'
                      '$status $body_bytes_sent "$http_referer"'
                      '"$http_user_agent" "$upstream_addr" $request_time';
···

#检测nginx配置文件是否有错
[root@minglinux-02 /usr/local/nginx/conf/vhost] /usr/local/nginx/sbin/nginx -t
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
[root@minglinux-02 /usr/local/nginx/conf/vhost] /usr/local/nginx/sbin/nginx -s reload

最后
130上curl '192.168.162.130:9200/_cat/indices?v'
检查是否有nginx-test开头的索引生成
如果有，才能到kibana里去配置该索引
左侧点击“Managerment”-> “Index Patterns”-> “Create Index Pattern”
Index pattern这里写nginx-test-*
之后点击左侧的Discover

#130上查看索引
[root@minglinux-01 ~] curl '192.168.162.130:9200/_cat/indices?v' 
health status index                 uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .kibana_1             IpJkISvGSBecrHJR5dNMtA   1   1          4            0     39.4kb         19.7kb
green  open   system-syslog-2019.03 cWR1XimDQE6PkexAIHCYkA   5   1       2887            0      2.2mb            1mb
green  open   nginx-test-2019.03.20 hVWCcygQRvSfmE54H_vyIg   5   1       2821            0      1.4mb        703.4kb

到kibana里去配置新索引
日志页面

[root@minglinux-02 /usr/share/logstash/bin] tail -f /tmp/elk_access.log
elk.ming.com 192.168.162.1 - - [21/Mar/2019:02:11:54 +0800] "GET /bundles/kibana.bundle.js HTTP/1.1"304 0 "http://elk.ming.com/app/kibana""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0" "192.168.162.130:5601" 0.038
elk.ming.com 192.168.162.1 - - [21/Mar/2019:02:11:59 +0800] "GET /api/console/api_server?sense_version=%40%40SENSE_VERSION&apis=es_6_0 HTTP/1.1"200 19139 "http://elk.ming.com/app/kibana""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0" "192.168.162.130:5601" 0.104
elk.ming.com 192.168.162.1 - - [21/Mar/2019:02:12:01 +0800] "GET /api/spaces/space HTTP/1.1"200 114 "http://elk.ming.com/app/kibana""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0" "192.168.162.130:5601" 0.095
elk.ming.com 192.168.162.1 - - [21/Mar/2019:02:12:01 +0800] "GET /api/saved_objects/_find?type=index-pattern&fields=title&search=*&search_fields=title&per_page=1 HTTP/1.1"200 179 "http://elk.ming.com/app/kibana""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0" "192.168.162.130:5601" 0.091
elk.ming.com 192.168.162.1 - - [21/Mar/2019:02:12:01 +0800] "GET /bundles/ebdca7741674eca4e1fadeca157f3ae6.svg HTTP/1.1"304 0 "http://elk.ming.com/bundles/commons.style.css""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0" "192.168.162.130:5601" 0.193
elk.ming.com 192.168.162.1 - - [21/Mar/2019:02:12:01 +0800] "GET /api/security/v1/me HTTP/1.1"200 2 "http://elk.ming.com/app/kibana""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0" "192.168.162.130:5601" 0.175
elk.ming.com 192.168.162.1 - - [21/Mar/2019:02:12:01 +0800] "GET /ui/fonts/open_sans/open_sans_v15_latin_regular.woff2 HTTP/1.1"304 0 "http://elk.ming.com/app/kibana""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0" "192.168.162.130:5601" 0.224
elk.ming.com 192.168.162.1 - - [21/Mar/2019:02:12:01 +0800] "GET /api/xpack/v1/info HTTP/1.1"200 679 "http://elk.ming.com/app/kibana""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0" "192.168.162.130:5601" 0.079
elk.ming.com 192.168.162.1 - - [21/Mar/2019:02:12:02 +0800] "GET /ui/fonts/open_sans/open_sans_v15_latin_700.woff2 HTTP/1.1"200 14720 "http://elk.ming.com/app/kibana""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0" "192.168.162.130:5601" 0.157
elk.ming.com 192.168.162.1 - - [21/Mar/2019:02:12:02 +0800] "GET /ui/fonts/open_sans/open_sans_v15_latin_600.woff2 HTTP/1.1"200 14544 "http://elk.ming.com/app/kibana""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0" "192.168.162.130:5601" 0.161

十一、使用beats采集日志

https://www.elastic.co/cn/products/beats
filebeat metricbeat packetbeat winlogbeat auditbeat heartbeat
可扩展，支持自定义构建
在128上执行
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.0.0-x86_64.rpm
rpm -ivh filebeat-6.0.0-x86_64.rpm
首先编辑配置文件
vim /etc/filebeat/filebeat.yml //增加或者更改
filebeat.prospectors:
- type: log
paths:
- /var/log/messages
output.console:
enable: true
/usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml //可以在屏幕上看到对应的日志信息
再编辑配置文件
vim /etc/filebeat/filebeat.yml //增加或者更改
filebeat.prospectors:
- input_type: log
paths:
- /var/log/elasticsearch/minglinux.log
output.elasticsearch:
hosts: ["192.168.128.130:9200"]
systemctl start filebeat

[root@minglinux-03 ~] wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.0.0-x86_64.rpm
[root@minglinux-03 ~] rpm -ivh  filebeat-6.0.0-x86_64.rpm
准备中...                          ################################# [100%]
正在升级/安装...
   1:filebeat-6.0.0-1                 ################################# [100%]

#修改配置文件
[root@minglinux-03 ~] vim /etc/filebeat/filebeat.yml 
#修改或新增如下内容
15 filebeat.prospectors:
 21 - type: log
 24 # enabled: false
 27   paths:
 28     - /var/log/messages
141 output.console:
142   enable: true
143 #-------------------------- Elasticsearch output ------------------------------
144 #output.elasticsearch:

#远程登录
[root@minglinux-02 ~] ssh minglinux-03
root@minglinux-03's password: 
Last login: Thu Mar 21 02:30:37 2019 from minglinux-02
[root@minglinux-03 ~] 登出
Connection to minglinux-03 closed.

#屏幕可以显示对应的日志信息
^C[root@minglinux-03 ~] /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml 
···
···
{"@timestamp":"2019-03-20T18:35:59.219Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.0.0"},"source":"/var/log/messages","offset":13019,"message":"Mar 21 02:35:56 minglinux-03 systemd: Starting Session 73 of user root.","prospector":{"type":"log"},"beat":{"version":"6.0.0","name":"minglinux-03","hostname":"minglinux-03"}}
{"@timestamp":"2019-03-20T18:35:59.219Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.0.0"},"prospector":{"type":"log"},"beat":{"name":"minglinux-03","hostname":"minglinux-03","version":"6.0.0"},"source":"/var/log/messages","offset":13084,"message":"Mar 21 02:35:57 minglinux-03 systemd-logind: Removed session 73."}

#再编辑配置文件

#增加或者更改如下内容
[root@minglinux-03 ~] vim /etc/filebeat/filebeat.yml 
···
 27   paths:
 28     - /var/log/elasticsearch/minglinux.log  #es的日志

141 #output.console:
142 #  enable: true

144 output.elasticsearch:
145   # Array of hosts to connect to.
146   hosts: ["192.168.162.130:9200"]
···

#启动filebeat
[root@minglinux-03 ~] systemctl start  filebeat
[root@minglinux-03 ~] ps aux |grep filebeat
root      21859 12.5  1.2 287348 12112 ?        Ssl  02:50   0:00 /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat
root      21867  0.0  0.0 112720   984 pts/0    S+   02:50   0:00 grep --color=auto filebeat

#查看索引
[root@minglinux-01 ~] curl '192.168.162.130:9200/_cat/indices?v' 
health status index                     uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .kibana_1                 IpJkISvGSBecrHJR5dNMtA   1   1          5            0     52.8kb         26.4kb
green  open   system-syslog-2019.03     cWR1XimDQE6PkexAIHCYkA   5   1      16469            0        7mb          3.5mb
green  open   filebeat-6.0.0-2019.03.21 WKCdeKzARK2npJwjkgHFtA   3   1          9            0     48.6kb         21.6kb
green  open   nginx-test-2019.03.20     hVWCcygQRvSfmE54H_vyIg   5   1      16403            0      6.5mb          3.5mb

到kibana里去配置新索引
日志页面

十二、扩展部分

x-pack 收费，免费 http://www.jianshu.com/p/a49d93212eca
https://www.elastic.co/subscriptions
Elastic stack演进 http://70data.net/1505.html
基于kafka和elasticsearch，linkedin构建实时日志分析系统 http://t.cn/RYffDoE
使用redis http://blog.lishiming.net/?p=463
ELK+Filebeat+Kafka+ZooKeeper 构建海量日志分析平台 https://www.cnblogs.com/delgyd/p/elk.html
http://www.jianshu.com/p/d65aed756587

ELK

目录

一、ELK介绍

二、ELK安装准备工作

三、安装es

四、配置es

五、curl查看es

六、安装kibanan

七、安装logstash

八、配置logstash

九、kibanan上查看日志

十、收集nginx日志

十一、使用beats采集日志

十二、扩展部分

相关文章

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读

ELK

目录

一、ELK介绍

二、ELK安装准备工作

三、安装es

四、 配置es

五、curl查看es

六、安装kibanan

七、安装logstash

八、配置logstash

九、kibanan上查看日志

十、收集nginx日志

十一、使用beats采集日志

十二、扩展部分

相关文章

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读

四、配置es