cassandra 日志中一直出现Connection rese

cassandra 日志中一直出现Connection reset by peer

cassandra 上阿里云后，启动cassandra日志中一直出现Connection reset by peer，更具体的日志请看下面

INFO  [epollEventLoopGroup-2-42] 2019-05-22 16:23:46,079 Message.java:623 - Unexpected exception during request; channel = [id: 0x107e01cb, L:/172.19.78.33:9042 - R:/100.120.134.66:28963]
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peer
        at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
INFO  [epollEventLoopGroup-2-43] 2019-05-22 16:23:46,583 Message.java:623 - Unexpected exception during request; channel = [id: 0xadc8d2f9, L:/172.19.78.33:9042 - R:/100.120.134.2:54244]
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peer
        at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
INFO  [epollEventLoopGroup-2-44] 2019-05-22 16:23:46,717 Message.java:623 - Unexpected exception during request; channel = [id: 0x2734761d, L:/172.19.78.33:9042 - R:/100.121.206.192:42460]
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peer
        at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
INFO  [epollEventLoopGroup-2-45] 2019-05-22 16:23:47,075 Message.java:623 - Unexpected exception during request; channel = [id: 0x9d1a7792, L:/172.19.78.33:9042 - R:/100.121.206.0:31693]
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peer
        at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
INFO  [epollEventLoopGroup-2-46] 2019-05-22 16:23:47,519 Message.java:623 - Unexpected exception during request; channel = [id: 0x9ef064a7, L:/172.19.78.33:9042 - R:/100.120.134.131:30985]
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peer
        at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
INFO  [epollEventLoopGroup-2-47] 2019-05-22 16:23:47,787 Message.java:623 - Unexpected exception during request; channel = [id: 0x72d74006, L:/172.19.78.33:9042 - R:/100.121.206.64:52459]
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peer
        at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
INFO  [epollEventLoopGroup-2-48] 2019-05-22 16:23:47,865 Message.java:623 - Unexpected exception during request; channel = [id: 0x703a2d37, L:/172.19.78.33:9042 - R:/100.120.134.195:19486]
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peer
        at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
INFO  [epollEventLoopGroup-2-49] 2019-05-22 16:23:48,111 Message.java:623 - Unexpected exception during request; channel = [id: 0xa43202af, L:/172.19.78.33:9042 - R:/100.121.206.129:33635]
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peer
        at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
INFO  [epollEventLoopGroup-2-50] 2019-05-22 16:23:48,136 Message.java:623 - Unexpected exception during request; channel = [id: 0x300cc413, L:/172.19.78.33:9042 - R:/100.120.134.66:10785]
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peer
        at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
INFO  [epollEventLoopGroup-2-51] 2019-05-22 16:23:48,614 Message.java:623 - Unexpected exception during request; channel = [id: 0xee01428d, L:/172.19.78.33:9042 - R:/100.120.134.2:40153]
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peer
        at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
INFO  [epollEventLoopGroup-2-52] 2019-05-22 16:23:48,644 Message.java:623 - Unexpected exception during request; channel = [id: 0x74db5764, L:/172.19.78.33:9042 - R:/100.121.206.192:40722]
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peer
        at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
INFO  [epollEventLoopGroup-2-53] 2019-05-22 16:23:49,155 Message.java:623 - Unexpected exception during request; channel = [id: 0x5205ed46, L:/172.19.78.33:9042 - R:/100.121.206.0:41350]
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peer
        at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]

这种情况在我们内网开发环境当然也发生过，但是都是配置的关系，导致别的cassandra节点不能连接，但是这次运维人员确认了配置没有问题。然后开始详细看错误提示，拿了其中一段日志出来分析

INFO  [epollEventLoopGroup-2-52] 2019-05-22 16:23:48,644 Message.java:623 - Unexpected exception during request; channel = [id: 0x74db5764, L:/172.19.78.33:9042 - R:/100.121.206.192:40722]
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peer
        at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]

这上面有两个明细的IP地址，一个是cassandra节点的，另一个不知道，然后我一开始并不知道 L跟R代表什么意思，后面去群里请教知道 L是local, R 是request， 这样的话就能理解这个日志了。就是有很多 100.121.206.* 的节点尝试跟 我们的cassandra 建立连接，然后一直导致出现建立连接失败，日志中这种错误信息一直在出现，增长率还蛮快的。

我们怀疑过是别的cassandra集群连接我们，我们修改了端口，但是还是有这个错误日志，最终排查到是因为做了负载均衡，一直在做连通性测试。