zookeeper连接太多

问题描述

某日，同事找到我，说他们的程序卡死了，不能连接zookeeper了。

错误情况

2019-03-27 14:28:56 - INFO org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:1032) : Opening socket connection to server 192.168.6.1/192.168.6.1:2181. Will not attempt to authenticate using SASL (unknown error)
2019-03-27 14:28:56 -ERROR org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:289) : Unable to open socket to 192.168.6.1/192.168.6.1:2181
2019-03-27 14:28:56 - WARN org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1162)   : Session 0x169bce56a6b043f for server null, unexpected error, closing socket connection and attempting reconnect
java.net.SocketException: 网络不可达
    at sun.nio.ch.Net.connect0(Native Method)
    at sun.nio.ch.Net.connect(Net.java:454)
    at sun.nio.ch.Net.connect(Net.java:446)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
    at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:277)
    at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:287)
    at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1024)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
2019-03-27 14:28:58 - INFO org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:1032) : Opening socket connection to server 192.168.6.1/192.168.6.1:2181. Will not attempt to authenticate using SASL (unknown error)

问题处理

1. 先排除是否zookeeper的问题，在这个机器上面，telnet 192.168.6.1 2181，发现不通
2. 登录zookeeper机器，使用bin/zkServer.sh status,查看zookeeper状态，发现zookeeper集群是正常的，那么zookeeper集群本身是没有问题的。
3. 查看zookeeper的日志（话说zookeeper的日志真够乱的，服务器上面有好几个zookeeper.out都不知道是哪个，只能一个个打开，通过最近的时间判别。），通过日志发现了问题的原因,如下：

2019-03-27 09:24:38,191 [myid:1] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of stream exception
2019-03-27 09:24:38,078 [myid:1] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /192.168.15.63 - max is 60
2019-03-27 09:24:38,003 [myid:1] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /192.168.15.63 - max is 60

日志说的很清楚，连接数太多了，超过了默认配置的60个连接数。

解决办法

方法一. 修改zookeeper默认的连接数，并重启zookeeper集群。

vi ZOOKEEPER_HOME/conf/zoo.cfg

image.png

注意：
maxClientCnxns=0，则代表不限制连接数。

方法二. 优化程序，减少zookeeper的连接数。