基于filebeat的elk日志平台部署 （版本都是7.X）

es搭建

java环境

由于es是自带绑定的java环境的，所有机器可以无需java环境。而如果你本来就有java环境，那么在启动的时会使用你的java环境，可能会不兼容。可以将es带的jdk作为java_home

搭建安装

​ es主机ip：192.168.157.10 、 192.168.157.20

官方搭建文档 https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-install.html

sysctl -w vm.max_map_count=262144          ###临时设置
vim /etc/sysctl.conf
###############################
vm.max_map_count=262144               ###给es用户262144的内存，永久设置 否则后面会报错
####################################
curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.2.0-linux-x86_64.tar.gz
tar -xf elasticsearch-7.2.0.tar.gz 
vim /etc/hosts
#######################
192.168.157.10 es1
192.168.157.20 es2
#######################
vim elasticsearch-7.2.0/config/elasticsearch.yml
#############################################################
cluster.name: elasticsearch
node.name: "node-2"
node.master: true
node.data: true
##数据存储地址
path.data: /data/apps/data
network.host: 192.168.157.20
##tcp数据传输端口
transport.tcp.port: 9300
#http对外端口
http.port: 9200
##是否开启http对外服务
http.cors.enabled: true
http.cors.allow-origin: "*"         ###允许header进行访问
##ping链接超时时间
discovery.zen.ping_timeout: 10s
discovery.seed_hosts:       ######es集群
  - 192.168.157.10:9300
  - 192.168.157.20:9300
cluster.initial_master_nodes:    #####master推选
  - node-1
  - node-2
###############################################################
mkdir /data/apps/data
useradd es
chown -R es.es data
chown -R es.es /data/apps/elasticsearch-7.2.0
echo "es ALL = (root) NOPASSWD:ALL" > /etc/sudoers.d/es
su - es
<<<<<<<<<<<<<<<<<<<<
cd /data/apps/elasticsearch-7.2.0/bin
./elasticsearch -d
<<<<<<<<<<<<<<<<<<<<<<
curl 192.168.157.20:9200  ###验证

head插件安装

需要环境 node.js   npm   grunt

wget https://nodejs.org/dist/v6.10.2/node-v6.10.2-linux-x64.tar.xz
tar -xvJf node-v10.16.0-linux-x64.tar.xz        ####node.js自带npm环境
vim /etc/profile
#############################
export NODE_HOME=/usr/local/node
export PATH=$PATH:$NODE_HOME/bin
##############################
. /etc/profile
 git clone https://github.com/mobz/elasticsearch-head.git  ##下载header插件
 chown -R es.es elasticsearch-head/
 cd elasticsearch-head/   ##一定要进这个目录
 npm install -g grunt --registry=https://registry.npm.taobao.org  ##grunt环境
 npm install 
 npm install grunt --save
 vim Gruntfile.js
 ######################################
 connect: {
                        server: {
                                options: {
                                        port: 9100,
                                        hostname: '0.0.0.0',  ##加这一行
                                        base: '.',
                                        keepalive: true
                                }
                        }

 #######################################
 ----修改 _site/app.js里this.base_uri = 里面 http://localhost:9200字段为本机ES端口与IP
 ------------------
 grunt server  #依旧在elasticsearch-head/那个目录下  
 
 访问 ip:9100

logstash搭建

需要java环境

redis搭建

tar -xf redis-3.2.11.tar.gz
cd redis-3.2.11
make && make install PREFIX=/data/apps/redis
cd /data/apps/redis
mkdir conf && cd conf
cp /tmp/redis-3.2.11/redis.conf  .
vim ./redis.conf
---------------------
daemonize yes
port 6380       ## 服务端口
bind 127.0.0.1 本机IP     ## 绑定地址
requirepass SHUIge@2013COM.   ##密码
----------------------------
useradd redis
chown -R redis.redis /data/apps/redis
su - redis
cd /data/apps/redis/bin
./redis-server ../conf/redis.conf  ########6380端口

logstash搭建部署

下载地址 https://www.elastic.co/cn/downloads/logstash

wget https://artifacts.elastic.co/downloads/logstash/logstash-7.2.0.tar.gz
tar -xf logstash-7.2.0.tar.gz
ln -s  /data/apps/logstash-7.2.0/bin/logstash /usr/bin/
 mkdir -p /etc/logstash/conf.d
 cd /etc/logstash/conf.d
 vim redis.conf
 ###################################################
input {
    redis {
        data_type => "list"
        db => 0
        key => "filebeat"
        host => "localhost"
        port => 6380
        password => "Redis@Passwd1"
        threads => 1
    }
}
output {
    stdout {
      codec => rubydebug
    }
    elasticsearch {
      hosts => ["192.168.157.10:9200","192.168.157.20:9200"]
      index => ["message-%{[fields][ip_host]}-%{+YYYY.MM.dd}"]  ###这里%{[fields][ip_host]}在filebeat里定义了
    }
}
 #####################################################
logstash -f /etc/logstash/conf.d/redis.conf &

filebeat搭建

下载地址 https://www.elastic.co/cn/downloads/beats/filebeat

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.2.0-linux-x86_64.tar.gz
tar -xf filebeat-7.2.0-linux-x86_64.tar.gz
vim filebeat-7.2.0-linux-x86_64/filebeat.yml
######################################################
filebeat.inputs:
- type: log
  paths:
    - /var/log/secure
  encoding: utf-8
  fields:
    ip_host: 192.168.157.75      ###这里是自定义的变量，在logstash里引用作为index。
output.redis:
  hosts: ["192.168.157.30:6380"]  
  password: "Redis@Passwd1"
  key: "filebeat"  
  db: 0
  timeout: 5
######################################################
nohup filebeat-7.2.0-linux-x86_64/filebeat -e -c filebeat.yml &
要起多个filebeat的话，直接指定不同的配置文件就行了

Kibana搭建

下载地址 https://www.elastic.co/cn/downloads/kibana

wget https://artifacts.elastic.co/downloads/kibana/kibana-7.2.0-linux-x86_64.tar.gz
tar -xf kibana-7.2.0-linux-x86_64.tar.gz
cd kibana-7.2.0-linux-x86_64
vim ../config/kibana.yml
###################################################
server.port: 5601
server.host: "0.0.0.0"
server.name: "kibana"
elasticsearch.hosts: ["http://192.168.157.10:9200","http://192.168.157.20:9200"]
elasticsearch.preserveHost: true
elasticsearch.pingTimeout: 1500
elasticsearch.requestTimeout: 30000
logging.dest: /var/log/kibana/kibana.log
xpack.security.enabled: false   ######关闭安全模式 
i18n.locale: zh-CN       ###中文
###################################################
nohup ./kibana --allow-root -c ../config/kibana.yml &

关于错误

kibana

出现了kibana not ready ,看日志发现es没有响应。所有还是es有问题

es

因为版本过高，导致许多的es的配置文件里的许多参数不对。到官网查找文档后解决