连接zookeeper报错No route to host

最近在学习微服务，遇到了很多问题，今天又遇到了个连接zookeeper报错的问题，在此记录下。

学生课程管理系统简单微服务架构图：

image.png

各个服务容器化以后，用docker-compose启动，其中docker-compose.yml文件如下：

version: "3"

services:
  message-service:
    image: message-thrift-python-service:latest

  user-service:
    image: user-thrift-service:latest
    command:
    - "--mysql.address=192.168.1.103"

  user-edge-service:
    image: user-edge-service:latest
    links:
    - user-service
    - message-service
    command:
    - "--redis.address=192.168.1.103"

  course-service:
    image: course-service:latest
    links:
    - user-service
    command:
    - "--mysql.address=192.168.1.103"
    - "--zookeeper.address=192.168.1.103"

  course-edge-service:
    image: course-edge-service:latest
    links:
    - user-edge-service
    command:
    - "--zookeeper.address=192.168.1.103"

  api-gateway-zuul:
    image: api-gateway-zuul:latest
    links:
    - course-edge-service
    - user-edge-service
    ports:
    - 8080:8080

docker-compose up -d启动后，访问course-edge-service服务时报错，日志里打印如下错误：

2018-10-04 08:32:32.584  INFO 1 --- [168.1.103:2181)] org.apache.zookeeper.ClientCnxn          : Opening socket connection to server 192.168.1.103/192.168.1.103:2181. Will not attempt to authenticate using SASL (unknown error)
2018-10-04 08:32:33.590  WARN 1 --- [168.1.103:2181)] org.apache.zookeeper.ClientCnxn          : Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect

java.net.NoRouteToHostException: No route to host
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_181]
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_181]
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) ~[zookeeper-3.4.6.jar!/:3.4.6-1569965]
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) ~[zookeeper-3.4.6.jar!/:3.4.6-1569965]

于是先docker-compose down掉各服务，暂且单独启动course-edge-service单服务看是否同样报错：

docker run -ti course-edge-service:latest --zookeeper.address=192.168.1.103

启动后报同样的错。

看报错日志应该就是连不上zookeeper，在宿主机上curl zookeeper地址，是通的：

[root@localhost course-edge-service]# curl 192.168.1.103:2181
curl: (52) Empty reply from server

进入zookeeper容器内看zookeeper状态：

image.png

[root@localhost course-edge-service]# docker exec -ti c2803533c4fb bash
bash-4.4# cd bin/
bash-4.4# ./zk
zkCleanup.sh            zkEnv.cmd               zkServer.cmd            zkTxnLogToolkit.sh
zkCli.cmd               zkEnv.sh                zkServer.sh             
zkCli.sh                zkServer-initialize.sh  zkTxnLogToolkit.cmd     
bash-4.4# ./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: leader
bash-4.4#

zookeeper状态正常。

在course-edge-service的容器内curl zookeeper地址：

[root@localhost course-edge-service]# docker exec -ti 5c993840aeae bash
root@5c993840aeae:/# curl 192.168.1.103:2181
curl: (7) Failed to connect to 192.168.1.103 port 2181: No route to host
root@5c993840aeae:/# 
root@5c993840aeae:/# ping 192.168.1.103
PING 192.168.1.103 (192.168.1.103) 56(84) bytes of data.
64 bytes from 192.168.1.103: icmp_seq=1 ttl=64 time=0.255 ms
64 bytes from 192.168.1.103: icmp_seq=2 ttl=64 time=0.163 ms
64 bytes from 192.168.1.103: icmp_seq=3 ttl=64 time=0.164 ms

发现确实有问题。

解决问题：

查看防火墙状态：

$ systemctl status firewalld.service
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2018-10-04 00:38:32 CST; 15h ago
 Main PID: 826 (firewalld)
   Memory: 2.4M
   CGroup: /system.slice/firewalld.service
           └─826 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid

关闭防火墙：

$ systemctl stop firewalld.service

course-edge-service的容器内再curl zookeeper地址：

[root@localhost course-edge-service]# docker exec -ti 5c993840aeae bash
root@5c993840aeae:/# curl 192.168.1.103:2181
curl: (52) Empty reply from server
root@5c993840aeae:/# 

重新启动各个微服务，又报错如下：

[root@localhost micro-service]# docker-compose up -d
Creating network "micro-service_default" with the default driver
ERROR: Failed to Setup IP tables: Unable to enable SKIP DNAT rule:  (iptables failed: iptables --wait -t nat -I DOCKER -i br-07454d600e41 -j RETURN: iptables: No chain/target/match by that name.
 (exit status 1))

重启docker后解决：

[root@localhost micro-service]#systemctl restart docker
[root@localhost micro-service]# docker-compose up -d
Creating network "micro-service_default" with the default driver
Creating micro-service_user-service_1_11592de6bbc3    ... done
Creating micro-service_message-service_1_b225fa599858 ... done
Creating micro-service_user-edge-service_1_4f8b6edf7333 ... done
Creating micro-service_course-service_1_69e9bdcbd81e    ... done
Creating micro-service_course-edge-service_1_81d9188df304 ... done
Creating micro-service_api-gateway-zuul_1_e0bdf98eebef    ... done