Docker exec 出现 "fork/exec /proc/

docker 容器正常运行，但不能 exec

# docker ps | grep mq
5d0e262527cf        rabbitmq:3-management         "docker-entrypoint..."   12 months ago       Up 3 months         10.168.93.209:4369->4369/tcp, 10.168.93.209:5671-5672->5671-5672/tcp, 10.168.93.209:15671-15672->15671-15672/tcp, 10.168.93.209:25672->25672/tcp   mq01
# docker exec -it mq01 /bin/bash
rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:75: starting setns process caused \"fork/exec /proc/self/exe: no such file or directory\""

docker 版本信息

# docker version
Client:
 Version:         1.13.1
 API version:     1.26
 Package version: <unknown>
 Go version:      go1.8.3
 Git commit:      774336d/1.13.1
 Built:           Wed Mar  7 17:06:16 2018
 OS/Arch:         linux/amd64

Server:
 Version:         1.13.1
 API version:     1.26 (minimum version 1.12)
 Package version: <unknown>
 Go version:      go1.8.3
 Git commit:      774336d/1.13.1
 Built:           Wed Mar  7 17:06:16 2018
 OS/Arch:         linux/amd64
 Experimental:    false
# docker info
Containers: 20
 Running: 20
 Paused: 0
 Stopped: 0
Images: 215
Server Version: 1.13.1
Storage Driver: devicemapper
 Pool Name: docker-202:17-4703339-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 107.4 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 20.17 GB
 Data Space Total: 107.4 GB
 Data Space Available: 87.2 GB
 Metadata Space Used: 17.29 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.13 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Data loop file: /data/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /data/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.140-RHEL7 (2017-05-03)
Logging Driver: json-file
Cgroup Driver: systemd
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Init Binary: docker-init
containerd version:  (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: N/A (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: N/A (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  WARNING: You're not using the default seccomp profile
  Profile: /etc/docker/seccomp.json
Kernel Version: 3.10.0-693.21.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 16
Total Memory: 31.25 GiB
Name: docker02
ID: 74LF:KJUT:GI6B:VKVC:OGWX:GBIB:C3WQ:W2ON:Y54T:YHL3:5TBJ:ATUD
Docker Root Dir: /data/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Registry Mirrors:
 https://edavcczl.mirror.aliyuncs.com/
Live Restore Enabled: false
Registries: docker.io (secure)

具体解决过程

1、查找出 docker 的容器 id 5d0e262527cf
# docker ps | grep rabbitmq
5d0e262527cf        rabbitmq:3-management         "docker-entrypoint..."   12 months ago       Up 3 months         10.168.93.209:4369->4369/tcp, 10.168.93.209:5671-5672->5671-5672/tcp, 10.168.93.209:15671-15672->15671-15672/tcp, 10.168.93.209:25672->25672/tcp   mq01

2、根据 docker 容器 id 5d0e262527cf 找到对应的 libcontainerd 的运行pid 7309
# ps -ef|grep libcontainerd | grep 5d0e262527cf
root       7309   1136  0 May07 ?        00:00:09 /usr/bin/docker-containerd-shim-current 5d0e262527cf7c3c7f459104ac542a45b050d42817d07026e3ce0cd20b7c5459 /var/run/docker/libcontainerd/5d0e262527cf7c3c7f459104ac542a45b050d42817d07026e3ce0cd20b7c5459 /usr/libexec/docker/docker-runc-current

3、用 nsenter 进入 docker 容器 5d0e262527cf 的 namespace
# nsenter -m -t 7309 bash

4、查看 docker 容器 5d0e262527cf 的 DeviceName
# docker inspect --format='{{.GraphDriver.Data.DeviceName}}' 5d0e262527cf
docker-202:17-4703339-5d0e262527cf7c3c7f459104ac542a45b050d42817d07026e3ce0cd20b7c5459

5、
可以看到 docker 容器 5d0e262527cf 的 /data/docker/devicemapper/mnt/docker容器ID 这个目录不存在
# ll /data/docker/devicemapper/mnt/5d0e262527cf7c3c7f459104ac542a45b050d42817d07026e3ce0cd20b7c5459
ls: cannot access /data/docker/devicemapper/mnt/5d0e262527cf7c3c7f459104ac542a45b050d42817d07026e3ce0cd20b7c5459: No such file or directory

只看到 /data/docker/devicemapper/mnt/docker容器ID-init 这个目录，所以执行 docker exec 时，会报错 \"fork/exec /proc/self/exe: no such file or directory\"
# ll /data/docker/devicemapper/mnt/5d0e262527cf7c3c7f459104ac542a45b050d42817d07026e3ce0cd20b7c5459-init
total 0

6、对比正常 docker 容器 aa4416c1f1e8 的目录
# ll /data/docker/devicemapper/mnt/aa4416c1f1e8fb192e72b2cf60aae8507cc4bf7bbe69ef2b96d81e29640f7a4a
total 8
-rw-------  1 root root   64 Dec 11  2017 id
drwxr-xr-x 21 root root 4096 Apr  3 15:35 rootfs


7、创建 docker 容器 5d0e262527cf 对应的目录(去掉 -init 后缀)
# mkdir /data/docker/devicemapper/mnt/5d0e262527cf7c3c7f459104ac542a45b050d42817d07026e3ce0cd20b7c5459

8、重新挂载 docker 容器 5d0e262527cf 的 /data/docker/devicemapper/mnt/容器ID 目录
// 用法：mount /dev/mapper/docker容器的DeviceName -o rw,relatime,nouuid,attr2,inode64,sunit=512,swidth=1024,noquota -t xfs /Dockerd服务的数据目录/devicemapper/mnt/容器ID
# mount /dev/mapper/docker-202:17-4703339-5d0e262527cf7c3c7f459104ac542a45b050d42817d07026e3ce0cd20b7c5459 -o rw,relatime,nouuid,attr2,inode64,sunit=512,swidth=1024,noquota -t xfs /data/docker/devicemapper/mnt/5d0e262527cf7c3c7f459104ac542a45b050d42817d07026e3ce0cd20b7c5459

# ll /data/docker/devicemapper/mnt/5d0e262527cf7c3c7f459104ac542a45b050d42817d07026e3ce0cd20b7c5459
total 8
-rw-------  1 root root   64 Aug 10  2017 id
drwxr-xr-x 17 root root 4096 Dec 10  2017 rootfs

9、退出 docker 容器 5d0e262527cf 的 namespace
# exit
exit

10、测试此 docker 容器 5d0e262527cf 可以正常执行 docker exec
# docker exec -it 5d0e262527cf /bin/bash
root@mq01:/# ls
bin  boot  dev  docker-entrypoint.sh  etc  home  lib  lib32  lib64  libx32  media  mnt  opt  plugins  proc  root  run  sbin  srv  sys  tmp  usr  var
root@mq01:/# exit
exit

docker exec 出现问题时另一种折衷解决办法，通过 nsenter 进入容器

# docker exec -it 86ffcb615a74 /bin/bash
rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:75: starting setns process caused \"fork/exec /proc/self/exe: no such file or directory\""

# docker inspect -f {{.State.Pid}} 86ffcb615a74
6670
# nsenter -t 6670 -m -u -i -n -p
-bash: /var/log/usermonitor/usermonitor.log: No such file or directory
root@86ffcb615a74:/# ps -ef
UID         PID   PPID  C STIME TTY          TIME CMD
mysql         1      0  0 May07 ?        02:50:32 mysqld
root         60      0  0 14:32 ?        00:00:00 -bash
root         67     60  0 14:33 ?        00:00:00 ps -ef
-bash: /var/log/usermonitor/usermonitor.log: No such file or directory