ELK集群运行一段时间后，会发现当有大量日志传输时，kibana前端浏览器就会反应非常慢，查看系统CPU，发现Logstash和es所在进程此时占用CPU很高，我目前配置：
192.168.1.203 es- master logstash kibana
192.168.1.204 es-master
192.168.1.202 es-slave
添加redis后配置
192.168.1.203 es- master redis logstash kibana
192.168.1.204 es-master redis
192.168.1.202 es-slave redis
所以添加redis缓解压力
redis集群安装
注意：安装redis普通用户即可，但是升级ruby时需要root用户
集群安装可参考redis 集群安装
下面是我遇到的一些坑:
1 ruby安装redis依赖时报错
[elk@elk ~]$ gem install redis
Fetching: redis-4.1.2.gem (100%)
ERROR: Error installing redis:
redis requires Ruby version >= 2.3.0.
Centos7.2默认安装ruby2.0.0，所以需要先将rubu升级到2.3.0
升级ruby可参考
$ curl -L get.rvm.io | bash -s stable
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 194 100 194 0 0 249 0 --:--:-- --:--:-- --:--:-- 249
100 24535 100 24535 0 0 2398 0 0:00:10 0:00:10 --:--:-- 4191
Downloading https://github.com/rvm/rvm/archive/1.29.9.tar.gz
Downloading https://github.com/rvm/rvm/releases/download/1.29.9/1.29.9.tar.gz.asc
gpg: Signature made Wed 10 Jul 2019 04:31:02 PM CST using RSA key ID 39499BDB
gpg: Can't check signature: No public key
GPG signature verification failed for '/usr/local/rvm/archives/rvm-1.29.9.tgz' - 'https://github.com/rvm/rvm/releases/download/1.29.9/1.29.9.tar.gz.asc'! Try to install GPG v2 and then fetch the public key:
sudo gpg2 --keyserver hkp://pool.sks-keyservers.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
or if it fails:
command curl -sSL https://rvm.io/mpapis.asc | sudo gpg2 --import -
command curl -sSL https://rvm.io/pkuczynski.asc | sudo gpg2 --import -
In case of further problems with validation please refer to https://rvm.io/rvm/security
此时报错无法安装，根据提示执行
$ curl -sSL https://rvm.io/mpapis.asc | sudo gpg2 --import -
$ curl -sSL https://rvm.io/pkuczynski.asc | sudo gpg2 --import -
再执行,此时已安装成功
# curl -L get.rvm.io | bash -s stable
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 194 100 194 0 0 254 0 --:--:-- --:--:-- --:--:-- 254
100 24535 100 24535 0 0 1362 0 0:00:18 0:00:18 --:--:-- 1838
Downloading https://github.com/rvm/rvm/archive/1.29.9.tar.gz
Downloading https://github.com/rvm/rvm/releases/download/1.29.9/1.29.9.tar.gz.asc
gpg: Signature made Wed 10 Jul 2019 04:31:02 PM CST using RSA key ID 39499BDB
gpg: Good signature from "Piotr Kuczynski <piotr.kuczynski@gmail.com>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 7D2B AF1C F37B 13E2 069D 6956 105B D0E7 3949 9BDB
GPG verified '/usr/local/rvm/archives/rvm-1.29.9.tgz'
Creating group 'rvm'
Installing RVM to /usr/local/rvm/
Installation of RVM in /usr/local/rvm/ is almost complete:
* First you need to add all users that will be using rvm to 'rvm' group,
and logout - login again, anyone using rvm will be operating with `umask u=rwx,g=rwx,o=rx`.
* To start using RVM you need to run `source /etc/profile.d/rvm.sh`
in all your open shell windows, in rare cases you need to reopen all shell windows.
* Please do NOT forget to add your users to the rvm group.
The installer no longer auto-adds root or users to the rvm group. Admins must do this.
Also, please note that group memberships are ONLY evaluated at login time.
This means that users must log out then back in before group membership takes effect!
Thanks for installing RVM
Please consider donating to our open collective to help us maintain RVM.
Donate: https://opencollective.com/rvm/donate
查看配置文件并执行
$ find / -name rvm.sh
/etc/profile.d/rvm.sh
$ source /etc/profile.d/rvm.sh
$ rvm requirements
$ source /usr/local/rvm/scripts/rvm #刷新相关执行文件
$ rvm list known | grep ruby #查看ruby相关版本
安装ruby2.3.3
$ rvm install 2.3.3
Searching for binary rubies, this might take some time.
No binary rubies available for: centos/7/x86_64/ruby-2.3.3.
Continuing with compilation. Please read 'rvm help mount' to get more information on binary rubies.
Checking requirements for centos.
Requirements installation successful.
Installing Ruby from source to: /usr/local/rvm/rubies/ruby-2.3.3, this may take a while depending on your cpu(s)...
ruby-2.3.3 - #downloading ruby-2.3.3, this may take a while depending on your connection...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 13.7M 100 13.7M 0 0 27229 0 0:08:50 0:08:50 --:--:-- 28442
ruby-2.3.3 - #extracting ruby-2.3.3 to /usr/local/rvm/src/ruby-2.3.3.....
ruby-2.3.3 - #applying patch /usr/local/rvm/patches/ruby/ruby_2_3_gcc7.patch.
ruby-2.3.3 - #applying patch /usr/local/rvm/patches/ruby/2.3.3/random_c_using_NR_prefix.patch.
ruby-2.3.3 - #configuring..........................................................
ruby-2.3.3 - #post-configuration..
ruby-2.3.3 - #compiling..................................................................................
ruby-2.3.3 - #installing............................
ruby-2.3.3 - #making binaries executable..
ruby-2.3.3 - #downloading rubygems-3.0.4
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 862k 100 862k 0 0 20883 0 0:00:42 0:00:42 --:--:-- 16347
ruby-2.3.3 - #extracting rubygems-3.0.4.....
ruby-2.3.3 - #removing old rubygems........
ruby-2.3.3 - #installing rubygems-3.0.4.........................................
ruby-2.3.3 - #gemset created /usr/local/rvm/gems/ruby-2.3.3@global
ruby-2.3.3 - #importing gemset /usr/local/rvm/gemsets/global.gems................................................................
ruby-2.3.3 - #generating global wrappers.......
ruby-2.3.3 - #gemset created /usr/local/rvm/gems/ruby-2.3.3
ruby-2.3.3 - #importing gemsetfile /usr/local/rvm/gemsets/default.gems evaluated to empty gem list
ruby-2.3.3 - #generating default wrappers.......
ruby-2.3.3 - #adjusting #shebangs for (gem irb erb ri rdoc testrb rake).
Install of ruby-2.3.3 - #complete
Please be aware that you just installed a ruby that requires 2 patches just to be compiled on an up to date linux system.
This may have known and unaccounted for security vulnerabilities.
Please consider upgrading to ruby-2.6.3 which will have all of the latest security patches.
Ruby was built without documentation, to build it run: rvm docs generate-ri
将安装的ruby2.3.3设置为默认版本
rvm use 2.3.3 --default
Using /usr/local/rvm/gems/ruby-2.3.3
卸载旧版本
rvm remove 2.0.0
查看新版本
-bash-4.2$ ruby -v
ruby 2.3.3p222 (2016-11-21 revision 56859) [x86_64-linux]
再次安装redis依赖
[root@elk2 ~]# gem install redis
Fetching redis-4.1.2.gem
Successfully installed redis-4.1.2
Parsing documentation for redis-4.1.2
Installing ri documentation for redis-4.1.2
Done installing documentation for redis after 1 seconds
1 gem installed
切换到普通用户时报错
Last login: Fri Aug 9 10:41:29 CST 2019 on pts/0
ERROR: Missing proper 'which' command. Make sure it is installed before using RVM!
WARNING:
Errors sourcing '/usr/local/rvm/scripts/base'.
RVM will likely not work as expected.
查看了which的权限,普通用户不可访问，可能是ruby还需要which的权限吧，赋权即可
[root@elk2 ~]# ls -al /usr/bin/which
-rwx------. 1 root root 24336 Jun 10 2014 /usr/bin/which
[root@elk2 ~]# chmod 755 /usr/bin/which
[root@elk2 ~]# su - elk
Last login: Fri Aug 9 10:42:06 CST 2019 on pts/0
-bash-4.2$ ruby -v
ruby 2.3.3p222 (2016-11-21 revision 56859) [x86_64-linux]
附录
redis.conf参考
bind 192.168.1.202
protected-mode yes
port 7006
tcp-backlog 511
timeout 0
tcp-keepalive 300
daemonize yes
supervised no
pidfile /elk/redis/redis-4.0.1/redis-cluster/pid/redis_7006.pid
loglevel notice
logfile "/elk/redis/redis-4.0.1/redis-cluster/logs/redis_7006.log"
databases 16
always-show-logo yes
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /elk/redis/redis-4.0.1/redis-cluster/7006/
slave-serve-stale-data yes
slave-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
slave-priority 100
lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no
slave-lazy-flush no
appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
aof-use-rdb-preamble no
lua-time-limit 5000
cluster-enabled yes
cluster-config-file nodes-7006.conf
cluster-node-timeout 15000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes
启动
for((i=0;i<=2;i++)); do /elk/redis/redis-4.0.1/src/redis-server /elk/redis/redis-4.0.1/redis-cluster/700$i/redis.conf; done
for((i=3;i<=5;i++)); do /elk/redis/redis-4.0.1/src/redis-server /elk/redis/redis-4.0.1/redis-cluster/700$i/redis.conf; done
for((i=6;i<=8;i++)); do /elk/redis/redis-4.0.1/src/redis-server /elk/redis/redis-4.0.1/redis-cluster/700$i/redis.conf; done
关闭
for((i=0;i<=2;i++)); do /elk/redis/redis-4.0.1/src/redis-cli -c -h 192.168.1.204 -p 700$i shutdown;done
for((i=3;i<=5;i++)); do /elk/redis/redis-4.0.1/src/redis-cli -c -h 192.168.1.203 -p 700$i shutdown;done
for((i=6;i<=8;i++)); do /elk/redis/redis-4.0.1/src/redis-cli -c -h 192.168.1.202 -p 700$i shutdown;done
检查集群是否可用
/elk/redis/redis-4.0.1/src/redis-cli -h 192.168.1.204 -c -p 7000
/elk/redis/redis-4.0.1/src/redis-cli -h 192.168.1.203 -c -p 7003
/elk/redis/redis-4.0.1/src/redis-cli -h 192.168.1.202 -c -p 7006
检查集群状态（任意节点任意端口都可检查）
/elk/redis/redis-4.0.1/src/redis-trib.rb check 192.168.1.204:7000
列出集群节点
-bash-4.2$ /elk/redis/redis-4.0.1/src/redis-cli -h 192.168.1.204 -c -p 7000
192.168.1.204:7000> cluster nodes
打印集群信息
192.168.1.204:7000> cluster info
登录集群节点
/elk/redis/redis-4.0.1/src/redis-cli -h 192.168.1.203 -c -p 7003
redis清理缓存
redis缓存清理
1）登陆redis
/elk/redis/redis-4.0.1/src/redis-cli -h 192.168.1.203 -c -p 7003
2）执行：
192.168.1.100:6379> dbsize
3）
192.168.1.100:6379> flushall
================================================================
redis集群指定key值得缓存清理：
1)登录至指定端口的redis服务器
# src/redis-cli -c -h 192.168.1.100 -p 6379
2）查看所有key值 （或者使用info命令也能查看出来）
keys *
3）删除指定索引的值
del key
redis 命令行执行任务
$ /elk/redis/redis-4.0.1/src/redis-cli -h 192.168.1.202 -c -p 7006 cluster nodes|grep master
redis集群详解请参考
redis配置文件详解请参考
2 redis集群进程密码认证
登录
$ /elk/redis/redis-4.0.1/src/redis-cli -h 192.168.1.202 -c -p 7008
192.168.1.202:7008> config set masterauth Wupao++2019
OK
192.168.1.202:7008> config set requirepass Wupao++2019
OK
每个节点都要执行上面两条密码，并且密码要一致，执行完上面的操作后执行其他命令就没有权限了，如
192.168.1.202:7008> keys *
(error) NOAUTH Authentication required.
之后登录必须添加-a 参数
/elk/redis/redis-4.0.1/src/redis-cli -h 192.168.1.204 -c -p 7001 -a yourpassword
或者登陆后
127.0.0.1:7000> auth "yourpassword"
ELK添加redis之logstash和filebeat修改
架构变为filebeat-->redis集群-->logstash，需要修改filebeat的输出和logstash的输入值
1 filebeat配置
filebeat.inputs:
- type: log
enabled: true
paths:
- /opt/filebeat/logs/test.log
fields:
env: test
...............
...............
output.redis:
# The Logstash hosts
enabled: true
hosts: ["192.168.1.103:7000"]
key: "elk-redis"
db: 0
password: "your passwd"
.......................
........................
注意：
hosts: ["192.168.1.103:7000"] --redis集群中某一个客户端，如果日志提示MOVED到其它节点，需要修改hosts的IP和端口，注意记住这个IP和端口，在logstash端读取也要填写这个IP和端口和ket值，如问题3
key: "elk-redis" --redis的key，redis集群会利用它的算法将key放置在某一个客户端上
db: 0 --redis集群只可以select 0填写其它的会报错
password: "your passwd" --登录redis集群的名字
3 redis端查看数据
登录：
/elk/redis/redis-4.0.1/src/redis-cli -h 192.168.1.203 -c -p 7003 -a your_passwd
查看：
redis 127.0.0.1:7000[0]> keys * # 出现这个key了 说明fielebeat的数据已经传输到redis中了
1) "elk-redis"
redis 127.0.0.1:7000[0]> llen elk-redis ##查看list长度
(integer) 2002
redis 127.0.0.1:7000[0]> lindex elk-redis 0 #查看相关数据
3 logstash配置
input {
redis {
# batch_count => 1 #返回的事件数量，此属性仅在list模式下起作用,注意取消会报错，注意问题1
host => "192.168.1.103" #redis地址
port => "7000" #redis端口号
password => "your passwd" #如果有安全认证，此项为密码
data_type => "list" #logstash redis插件工作方式
db => "0" #redis数据库的编号
key => "elk-redis" #监听的键值
threads => "5" #启用线程数量
}
}
3 问题
1 logstash读取redis中数据，logstash日志中报错
[logstash.inputs.redis ] Redis connection problem {:exception=>#<Redis::CommandError: CROSSSLOT Keys in request don't hash to the same slot>}
解决：
刚开始的配置中
将batch_count 这个属性干掉，再重新启动就可以读取redis集群的消息了
2 filebeat传输到redis中，filebeat日志中报错
filebeat Failed to connect to redis): NOAUTH Authentication required
解决：
需要正确添加 password: "your passwd"参数，因为登录redis集群是需要密码的
output.redis:
# The Logstash hosts
enabled: true
hosts: ["192.168.1.103:7000"]
key: "elk-redis"
db: 0
password: "your passwd"
3 filebeat传输到redis集群中，filebeat日志报错
Failed to publish events: MOVED 10463 192.168.1.202:7006
解决：如果有多个key，redis 集群还会根据它的算法告诉你应该在那个slot下，redis cluster共有16384个槽位，如果正好在这个客户端执行，那就直接执行了，如果不是，就会报出MOVED error，告诉你应该将数据传输到那个客户端，你只要根据提示进行更改为正确的IP即可，如将IP更改为192.168.1.202就可以正确输出到redis集群了
大概就是这样了，redis已经添加了，继续观察！
4 kibana中的日志显示在毫秒内是无法进行排序的
网上查找是redis插件详解
因为logstash对接redis的配置中thread => 5导致，更改未thread => 1后还是出现这个问题
添加batch_count => 1这个参数后一直报错
[2019-08-14T16:26:55,822][WARN ][logstash.inputs.redis ] Redis connection problem {:exception=>#<Redis::CommandError: CROSSSLOT Keys in request don't hash to the same slot>}
[2019-08-14T16:26:55,808][WARN ][logstash.inputs.redis ] Redis connection problem {:exception=>#<Redis::CommandError: CROSSSLOT Keys in request don't hash to the same slot>}
这个问题暂时还没解决
参考：
https://www.elastic.co/cn/blog/just_enough_redis_for_logstash
https://www.cnblogs.com/xing901022/p/4864690.html
https://blog.51cto.com/liqingbiao/2177873
网友评论