ELK 一键安装脚本

Readme

• 使用此脚本前，需将其中的ip更改为自己的服务器ip。
• 将需要安装的tar包拷贝到指定文件夹，此脚本为ELK离线安装。
• 所需准备的tar包，并拷贝到/opt/elk_soft下（可自由指定，指定后更改以下脚本中software_dir属性值）
  • jdk-8u181-linux-x64.tar.gz
  • elasticsearch-6.5.1.tar.gz
  • logstash-6.5.1.tar.gz
  • kibana-6.5.1-linux-x86_64.tar.gz
• 此脚本为单节点部署ELK，如果是集群部署，需新建slave节点的部署脚本，并将该脚本中1-10的内容拷贝到新建的脚本。更改各脚本中discovery.zen.ping.unicast.hosts: ["192.168.127.131"]属性为集群的ip列表
• 系统环境centos7

一、Elasticsearch脚本

1.1 安装

• 新建脚本并写入以下内容

touch install-elasticsearch.sh

#!/bin/bash
#mail:maguichang@unicloud.com
#date:20190325
#Info:AutoInstall ELK scripts
#Software:elasticsearch-6.5.1/logstash-6.5.1/kibana-6.5.1
#System:centos7
clear
echo "######################################################"
echo "#          Auto Install ELK                          #"
echo "#          Press Ctrl + C to cancel                  #"
echo "######################################################"

# 建议将需要安装的tar包拷贝到software_dir,没有则新建，该路径可自定义
# 新建 /opt/ntp 文件夹，将ntp相关组件拷贝到该目录下，进行时间同步
software_dir="/opt/elk_soft"
# 1.安装依赖ntp
cd /opt/ntp
rpm -ivh autogen-libopts-5.18-5.el7.x86_64.rpm
rpm -ivh ntpdate-4.2.6p5-22.el7.centos.x86_64.rpm
rpm -ivh ntp-4.2.6p5-22.el7.centos.x86_64.rpm

# 2.集群时间同步,每台均需设置
ntpdate ntp2.aliyun.com && echo "*/10 * * * * root /usr/sbin/ntpdate
ntp1.aliyun.com" >> /etc/crontab && echo "*/10 * * * * root
/usr/sbin/ntpdate 1.cn.pool.ntp.org" >> /etc/crontab

echo "#########################################"
echo 'ntpdate setup completed'
echo "#########################################"

# 3.java环境安装
mkdir -p /opt/java
cd $software_dir
tar zxvf jdk-8u181-linux-x64.tar.gz -C /opt/java > /dev/null 2>&1
echo '# JAVA-8u181' >> /etc/profile
echo 'JAVA_HOME=/opt/java/jdk1.8.0_181' >> /etc/profile
echo 'JAVA_BIN=/opt/java/jdk1.8.0_181/bin' >> /etc/profile
echo 'PATH=$PATH:$JAVA_BIN' >> /etc/profile
echo 'CLASSPATH=$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar' >> /etc/profile
echo 'export JAVA_HOME JAVA_BIN PATH CLASSPATH' >> /etc/profile
source /etc/profile
echo "java is already installed,please open another session to test it "

# 4.系统安全配置
ENFORCE_NUM=`grep -c "SELINUX=enforcing" /etc/selinux/config`
if [ $ENFORCE_NUM -eq 1 ];then
{
sed -i "s#SELINUX=enforcing#SELINUX=disabled#g" /etc/selinux/config
>/dev/null 2>&1
setenforce 0
} && echo "selinux is disabled"
else
echo "selinux is already changed ,you need not do it again "
fi
#history_size
HISIZE_NUM=`grep -c "HISTSIZE=1000" /etc/profile`
if [ $HISIZE_NUM -eq 1 ];then 
{
sed -i "s#HISTSIZE=1000#HISTSIZE=6000#g" /etc/profile
echo "HISTSIZE is changed to 6000"
}
else
echo " HISTSIZE is already changed to 6000, you need not do it again"
fi
## onboot shutdown firewalld \NetworkManager
systemctl disable NetworkManager.service && systemctl disable firewalld 
#kernel about tcp
net_num=`grep -n "^net.ipv4" /etc/sysctl.conf | wc -l`
#echo $net_num
[ $net_num -eq 0 ] && {
echo 'net.ipv4.tcp_tw_reuse = 1 ' >> /etc/sysctl.conf
echo 'net.ipv4.tcp_tw_recycle = 1 ' >> /etc/sysctl.conf
echo 'net.ipv4.tcp_fin_timeout = 5' >> /etc/sysctl.conf
/sbin/sysctl -p > /dev/null 2>&1
} || echo " the tcp_kernel is already changed ,you need not do it again" 

useDNS_num=`grep -n "#UseDNS yes" /etc/ssh/sshd_config | wc -l`
if [ $useDNS_num -eq 0 ];then
echo "the ssh configuration UseDNS is already changed ,you need not do
it again"
else
sed -i 's/#UseDNS yes/UseDNS no/g' /etc/ssh/sshd_config
fi
systemctl restart sshd.service >/dev/null 2>&1 

GSSNUM=`grep -n "GSSAPIAuthentication no" /etc/ssh/sshd_config | wc -l`
if [ $GSSNUM -eq 0 ];then
echo "the ssh configuration GSSAPI is already changed ,you need not
do it again"
else
sed -i 's/GSSAPIAuthentication yes/GSSAPIAuthentication no/g'
/etc/ssh/sshd_config 

fi
HISFORMAT_line_number=`grep "HISTTIMEFORMAT" /etc/profile | wc -l `
if [ $HISFORMAT_line_number -eq 0 ];then
echo "HISTTIMEFORMAT='%F %T '" >> /etc/profile
source /etc/profile
echo "HISTTIMEFORMAT is changed justnow"
else
echo "HISTTIMEFORMAT is already changed,you need not do it again "
fi

#nofile limits
num_limits=`grep "soft nofile" /etc/security/limits.conf | wc -l `
#echo $num_limits
[ $num_limits -eq 0 ] && {
echo "* soft nofile 102400" >> /etc/security/limits.conf
echo "* hard nofile 102400" >> /etc/security/limits.conf
echo "* soft core unlimited" >> /etc/security/limits.conf
echo "ulimit -c unlimited" >> /etc/profile
} || {
echo " limit already finish，you need not do it again"
}

#delete tty
echo "console" > /etc/securetty
echo "vc/1" >> /etc/securetty
echo "tty1" >> /etc/securetty

#kernel about tcp
net_num=`grep -n "^net.ipv4" /etc/sysctl.conf | wc -l`
#echo $net_num
[ $net_num -eq 0 ] && {
echo 'net.ipv4.tcp_tw_reuse = 1 ' >> /etc/sysctl.conf
echo 'net.ipv4.tcp_tw_recycle = 1 ' >> /etc/sysctl.conf
echo 'net.ipv4.tcp_fin_timeout = 5' >> /etc/sysctl.conf
/sbin/sysctl -p > /dev/null 2>&1
} || echo "the tcp_kernel is already changed，you need not do it again " 

# del system useless users
num=`awk -F ":" '{print $1}' /etc/passwd | egrep
"adm|lp|sync|shutdown|halt|news|uucp|operator|games|gopher" | wc -l`
if [ $num -eq 0 ];then
printf "the useless number is %s , you need not to del again \n" $num
else
for i in `awk -F ":" '{print $1}' /etc/passwd | egrep
"adm|lp|sync|shutdown|halt|news|uucp|operator|games|gopher" `;do
userdel -r $i >/dev/null 2>&1
done
echo "...................................useless user is
deleted....................................................."
fi

# del system useless group
group_num=`awk -F ":" '{print $1}' /etc/group | egrep
"adm|lp|news|uucp|games|dip|pppusers|popusers|slipusers" | wc -l `
if [ $group_num -gt 0 ];then
for j in `awk -F ":" '{print $1}' /etc/group | egrep
"adm|lp|news|uucp|games|dip|pppusers|popusers|slipusers"`;do
groupdel $j >/dev/null 2>&1
done
echo "..................................uselsess group is
deleted...................................................."
else
printf "the useless group number is %s ,there is no need to del again\n"
$group_num
fi
echo "#######################################"
echo "System security setup completed"
echo "#######################################"

#5.关闭防火墙
systemctl stop firewalld
systemctl disable firewalld

#6.ulimit值修改
ulimit -l unlimited
ulimit -a

echo "es soft memlock unlimited" >> /etc/security/limits.conf
echo "es hard memlock unlimited" >> /etc/security/limits.conf
echo "ulimit -c unlimited" >> /etc/profile
echo "vm.max_map_count=655360" >> /etc/sysctl.conf
sysctl -p

echo "############################################"
echo "#########ulimit setup completed#############"
echo "############################################"

#7.安装elasticsearch
cd $software_dir
tar zxvf elasticsearch-6.5.1.tar.gz -C /opt
cd /opt
mv elasticsearch-6.5.1 elasticsearch
mkdir -p /opt/elasticsearch/data/
mkdir -p /opt/elasticsearch/log/

#8.新增es用户和用户组，并设置密码
useradd es && chown -R es:es /opt/elasticsearch/
echo "请输入es用户的密码:"
passwd es

#9.配置elasticsearch.yml
cat >>/opt/elasticsearch/config/elasticsearch.yml<<EOF
cluster.name: my-elk
node.name: elk-server
path.data: /opt/elasticsearch/data/
path.logs: /opt/elasticsearch/log/
network.host: 192.168.127.131
http.port: 9200
discovery.zen.ping.unicast.hosts: ["192.168.127.131"]
action.destructive_requires_name: true

EOF

echo "######################################################"
echo "####### elasticsearch.yml setup completed ############"
echo "######################################################"
#10.配置jvm.options
sed -i "s/Xms2g/Xms3g/g" /opt/elasticsearch/config/jvm.options
sed -i "s/Xmx2g/Xmx3g/g" /opt/elasticsearch/config/jvm.options
echo "######################################################"
echo "#           jvm.options setup completed              #"
echo "######################################################"

echo "##### elasticsearch installed #######"

运行脚本进行安装

source install-elasticsearch.sh

1.2 启动es的脚本

vi start-elasticearch.sh

• 写入如下内容

#!/bin/bash

# es start
su - es -c "/opt/elasticsearch/bin/elasticsearch -d"

1.3 停止es的脚本

vi stop-elasticsearch.sh

• 写入如下内容

#!/bin/bash
# stop elasticsearch

es_ps=`jps|grep Elasticsearch`
arr_es=($es_ps)
#echo "stoping Elasticsearch id:"${arr[0]}...
echo "stoping Elasticsearch ..."
kill -9 ${arr_es[0]}
echo 'stoped'

1.4 卸载es的脚本

#!/bin/bash
# remove elasticsearch
echo "removing elasticsearch..."
cd /opt
rm -rf elasticsearch
echo "elasticsearch removed"

二、Kibana脚本

2.1 安装

touch install-kibana.sh

• 写入以下内容

#!/bin/bash
# install kibana
echo "###############################################"
echo "#####     开始安装kibana            ###########"
echo "                                               "
echo "###############################################"

software_dir=/opt/elk_soft
cd $software_dir

tar zxvf kibana-6.5.1-linux-x86_64.tar.gz -C /opt
cd /opt
mv kibana-6.5.1-linux-x86_64 kibana-6.5.1
cat >>/opt/kibana-5.6.1/config/kibana.yml<<EOF
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://192.168.127.131:9200"
EOF
echo "#########kibana installed 安装完成#################"

安装kibana

source install-kibana.sh

2.2 启动kibana的脚本

vi start-kibana.sh

• 写入如下内容

#!/bin/bash

# start kibana
echo "starting kibana..."
#echo "press Enter to continue"
#nohup /opt/kibana-6.5.1/bin/kibana & >/dev/null
/opt/kibana-6.5.1/bin/kibana &
echo "kibana started"

2.3 停止kibana的脚本

vi stop-kibana.sh

• 写入如下内容

#!/bin/bash

#stop kibana
echo "kibana stoping ..."
ps -ef | netstat -anp | grep 5601 | awk '{printf $7}' | cut -d / -f 1 | xargs kill
echo "kibana stoped"

2.4. 卸载kibana的脚本

vi remove-kibana.sh

• 写入如下内容

#!/bin/bash

# remove kibana
cd /opt
rm -rf kibana-6.5.1

三、Logstash脚本

3.1 安装

vi install-logstash.sh

• 写入以下命令

#!/bin/bash

# install logstash 安装logstash
echo "##########################################"
echo "#######开始安装logstash###################"
echo "##########################################"

software_dir="/opt/elk_soft"
cd $software_dir
tar zxvf logstash-6.5.1.tar.gz -C /opt
mkdir -p /opt/logstash-5.6.1/conf.d
cat>>/opt/logstash-6.5.1/conf.d/test.conf<<EOF
input{
    stdin{}
}
output{
    elasticsearch{
        hosts => "192.168.127.131:9200"
        index => "mytest-%{+YYYY.MM.dd}"
     }
     stdout { codec =>rubydebug}
}
EOF

echo "##########################################"
echo "#####logstash安装完成#####################"
echo "##########################################"

安装运行以下命令

source install-logstash.sh

3.2 启动logstash测试

 cd /opt/logstash-5.6.1
 bin/logstash -f /opt/logstash-6.5.1/conf.d/test.conf 
 # 控制台输入任意字符，回车

访问服务器ip的5601端口

以测试机192.168.127.131为例，访问其5601端口。

创建索引mytest-*，如果在discover可收到logstash控制台输入的字符，说明elk服务搭建、启动成功。

3.3停止logstash的脚本

vi stop-logstash.sh

• 写入以下内容

#!/bin/bash
# stop logstash
echo "stoping logstash..."
logstash_ps=`ps -ef|grep logstash|awk '{print $2}'`
arr_ps=($logstash_ps)
echo ${arr_ps[0]}
kill -9 ${arr_ps[0]}
echo "logstash stoped"

3.4 卸载logstash的脚本

vi remove-logstash.sh

• 写入以下内容

# remove logstash
echo "removing logstash ..."
rm -rf logstash-6.5.1
echo "logstash removed"