更多关注 http://www.mknight.cn/post/615/
Elasticsearch-Logstash-Kibana（一）环境搭建
Elasticsearch-Logstash-Kibana（二）数据可视化
Elasticsearch-Logstash-Kibana（三）环配置优化
安装
环境依赖
To check your Java version, run the following command:
java -version
On systems with Java installed, this command produces output similar to the following:
java version "1.8.0_65"
Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)
简单说，当前elk版本只支持jdk 8.0版本。
导入源
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
新建repo文件
Add the following in your /etc/yum.repos.d/ directory in a file with a .repo suffix, for example logstash.repo
[logstash-6.x]
name=Elastic repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
yum 安装
And your repository is ready for use. You can install it with:
sudo yum install elasticsearch logstash kibana
elasticsearch
修改配置文件
vim /etc/elasticsearch/elasticsearch.yml
network.host: 192.168.1.40
#如果是6.0系统的Linux，需要加下面两行
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
启动
/etc/init.d/elasticsearch start
访问该IP端口，检查是否正常
image
gen
Kibana
修改配置文件
vim /etc/kibana/kibana.yml
elasticsearch.url: "http://192.168.1.40:30036"
server.host: "0.0.0.0"
启动
/etc/init.d/kibana start
logstash
建立软链
ln -s /etc/logstash/ /usr/share/logstash/config
建立配置文件
vim /usr/share/logstash/config/conf.d/elk.conf
input {
file {
path => [ "/var/log/nginx/access.log" ]
start_position => "beginning"
ignore_older => 0
}
}
filter {
grok {
match => { "message" => "%{NGINXACCESS}" }
}
geoip {
source => "http_x_forwarded_for"
target => "geoip"
database => "/usr/share/logstash/plugin/GeoLite2-City.mmdb"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float" ]
convert => [ "response","integer" ]
convert => [ "bytes","integer" ]
replace => { "type" => "nginx_access" }
remove_field => "message"
}
useragent {
source => "agent"
target => "device"
}
date {
match => [ "timestamp","dd/MMM/yyyy:HH:mm:ss Z"]
}
mutate {
remove_field => "timestamp"
}
}
output {
elasticsearch {
hosts => ["192.168.1.40:30005"]
index => "logstash-mknight-nginx-%{+YYYY.MM.dd}"
#以下6.0已经不支持
#flush_size => 100
sniffing => true
}
stdout {codec => rubydebug}
}
配置GeoIP
mkdir plugin
cd plugin
wget http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz
gzip -d GeoLite2-City.mmdb.gz
配置nginx变量
mkdir patterns
vim patterns/nginx
NGUSERNAME [a-zA-Z\.\@\-\+_%]+
NGUSER %{NGUSERNAME}
NGINXACCESS %{IPORHOST:clientip} - %{NOTSPACE:remote_user} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} \"%{IPV4:http_x_forwarded_for}\"
启动
./bin/logstash -f config/conf.d/elk.conf
检查日志启动情况
tail -100 /var/log/logstash/logstash-plain.log
访问Kibana
建立索引
访问机器5601端口，设置index pattern，点击 create
image
配置Discover
点击 Discover 就可以看到导入的日志记录
image
另外，点击Avaliable Fields下面的字段，选择Add，就可以展示在记录中
image
这样elk的基本配置就完成了，后面的文章里说一下如何配置kibana的图表已经Dashboard。
问题整理：
Kibana
Kibana IP端口无法访问：
# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
#server.host: "localhost"
server.host: "0.0.0.0"
elasticsearch
elasticsearch 无法更改IP
image
问题原因：因为Centos6不支持SecComp，而ES5.2.1默认bootstrap.system_call_filter为true进行检测，所以导致检测失败，失败后直接导致ES不能启动。详见 ：https://github.com/elastic/elasticsearch/issues/22899
解决方法：在elasticsearch.yml中配置bootstrap.system_call_filter为false，注意要在Memory下面:
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
max number of threads [2048] for user [elasticsearch] is too low, increase to at least [4096]
[1]: max number of threads [2048] for user [elasticsearch] is too low, increase to at least [4096]
[2]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
解决：
vi /etc/security/limits.conf
添加如下内容:
* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096
Could not find logstash.yml
问题
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs to console
解决
cd /usr/share/logstash
ln -s /etc/logstash ./config
设置配置文件实时生效，从而不用频繁地启停Logstash。修改/etc/logstash/logstash.yml：
config.reload.automatic: true
内存不足
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000c5330000, 181207040, 0) failed; error='Cannot allocate memory'
内存不足： 减少启动程序所需内存，或加大内存，如关闭一些程序。
GeoIP
mkdir plugin
cd plugin
wget http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz
gzip -d GeoLite2-City.mmdb.gz
网友评论