问题
这两天线上碰到docker服务无法启动的线上问题，所有的docker命令都失败。
原因分析
底层docker storage driver对接的是zfs，通过查看docker相关的log发现zfs相关的命令都失败了，如下所示：
Jul 27 09:53:45 hostxx dockerd[20333]: time="2021-07-27T09:53:45.998861018Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jul 27 09:53:46 hostxx dockerd[20333]: time="2021-07-27T09:53:46.011060514Z" level=info msg="[graphdriver] using prior storage driver: zfs"
Jul 27 09:53:46 hostxx dockerd[20333]: time="2021-07-27T09:53:46.366724336Z" level=warning msg="Your kernel does not support swap memory limit"
Jul 27 09:53:46 hostxx dockerd[20333]: time="2021-07-27T09:53:46.366782436Z" level=warning msg="Your kernel does not support CPU realtime scheduler"
Jul 27 09:53:46 hostxx dockerd[20333]: time="2021-07-27T09:53:46.367561132Z" level=info msg="Loading containers: start."
Jul 27 09:53:46 hostxx dockerd[20333]: time="2021-07-27T09:53:46.510906642Z" level=warning msg="Failed to unmount 54a45a2bb6dfea5836a27ef335026cf6f8c0fd0acd77f22aabd58ce726ee43ff mount /var/lib/docker/zfs/graph/54a45a2bb6dfea5836a27ef335026cf6f8c0fd0acd77f22aabd58ce726ee43ff: invalid argument" storage-driver=zfs
Jul 27 09:53:46 hostxx dockerd[20333]: time="2021-07-27T09:53:46.564286741Z" level=error msg="Error unmounting /var/lib/docker/zfs/graph/54a45a2bb6dfea5836a27ef335026cf6f8c0fd0acd77f22aabd58ce726ee43ff: invalid argument" storage-driver=zfs
Jul 27 09:53:46 hostxx dockerd[20333]: time="2021-07-27T09:53:46.564388502Z" level=warning msg="failed to mount container to get BaseFs path" container=7ec3e2fbb1b340d989d92b8240d99891caa1119acdeb93227a643bbb7fcde52e error="error creating zfs mount: mount rpool/var/lib/docker/54a45a2bb6dfea5836a27ef335026cf6f8c0fd0acd77f22aabd58ce726ee43ff:/var/lib/docker/zfs/graph/54a45a2bb6dfea5836a27ef335026cf6f8c0fd0acd77f22aabd58ce726ee43ff: no such file or directory" paused=false restarting=false running=false
Jul 27 09:53:47 hostxx dockerd[20333]: time="2021-07-27T09:53:47.511680376Z" level=info msg="Removing stale sandbox 8f7c1358c387a553e891cad864755c45b4397aed5c41be131d0e9c3ac760a2f3 (7ec3e2fbb1b340d989d92b8240d99891caa1119acdeb93227a643bbb7fcde52e)"
Jul 27 09:53:47 hostxx dockerd[20333]: time="2021-07-27T09:53:47.556261341Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 6f2f85e2e72b7cdfed31f5f92d57338385f204953dc68fcbb49315b3bb9f4693 791a8351b6f3e680b8a78b7ec78d536eff896607cf8c2f95114d1b4cbbb16dac], retrying...."
Jul 27 09:53:47 hostxx dockerd[20333]: time="2021-07-27T09:53:47.783118834Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Jul 27 09:53:48 hostxx dockerd[20333]: time="2021-07-27T09:53:48.130441177Z" level=info msg="Loading containers: done."
Jul 27 09:53:48 hostxx dockerd[20333]: time="2021-07-27T09:53:48.320319246Z" level=info msg="Docker daemon" commit="20.10.2-0ubuntu1~18.04.3" graphdriver(s)=zfs version=20.10.2
Jul 27 09:53:48 hostxx dockerd[20333]: time="2021-07-27T09:53:48.320785965Z" level=info msg="Daemon has completed initialization"
Jul 27 09:53:48 hostxx systemd[1]: Started Docker Application Container Engine.
Jul 27 09:53:48 hostxx dockerd[20333]: time="2021-07-27T09:53:48.428254757Z" level=info msg="API listen on /var/run/docker.sock"
Jul 27 10:19:15 hostxx dockerd[20333]: time="2021-07-27T10:19:15.756820071Z" level=error msg="Error unmounting /var/lib/docker/zfs/graph/54a45a2bb6dfea5836a27ef335026cf6f8c0fd0acd77f22aabd58ce726ee43ff: invalid argument" storage-driver=zfs
Jul 27 10:19:15 hostxx dockerd[20333]: time="2021-07-27T10:19:15.805929948Z" level=warning msg="error locating sandbox id 8f7c1358c387a553e891cad864755c45b4397aed5c41be131d0e9c3ac760a2f3: sandbox 8f7c1358c387a553e891cad864755c45b4397aed5c41be131d0e9c3ac760a2f3 not found"
Jul 27 10:19:15 hostxx dockerd[20333]: time="2021-07-27T10:19:15.807043244Z" level=warning msg="Failed to unmount 54a45a2bb6dfea5836a27ef335026cf6f8c0fd0acd77f22aabd58ce726ee43ff mount /var/lib/docker/zfs/graph/54a45a2bb6dfea5836a27ef335026cf6f8c0fd0acd77f22aabd58ce726ee43ff: invalid argument" storage-driver=zfs
Jul 27 10:19:15 hostxx dockerd[20333]: time="2021-07-27T10:19:15.808954834Z" level=error msg="7ec3e2fbb1b340d989d92b8240d99891caa1119acdeb93227a643bbb7fcde52e cleanup: failed to delete container from containerd: no such container"
Jul 27 10:19:15 hostxx dockerd[20333]: time="2021-07-27T10:19:15.809323268Z" level=error msg="Handler for POST /v1.41/containers/7ec3e2fbb1b3/start returned error: error creating zfs mount: mount rpool/var/lib/docker/54a45a2bb6dfea5836a27ef335026cf6f8c0fd0acd77f22aabd58ce726ee43ff:/var/lib/docker/zfs/graph/54a45a2bb6dfea5836a27ef335026cf6f8c0fd0acd77f22aabd58ce726ee43ff: no such file or directory"
Jul 27 10:31:36 hostxx dockerd[20333]: time="2021-07-27T10:31:36.102865064Z" level=error msg="Error unmounting /var/lib/docker/zfs/graph/54a45a2bb6dfea5836a27ef335026cf6f8c0fd0acd77f22aabd58ce726ee43ff: invalid argument" storage-driver=zfs
Jul 27 10:31:36 hostxx dockerd[20333]: time="2021-07-27T10:31:36.197225096Z" level=warning msg="error locating sandbox id 8f7c1358c387a553e891cad864755c45b4397aed5c41be131d0e9c3ac760a2f3: sandbox 8f7c1358c387a553e891cad864755c45b4397aed5c41be131d0e9c3ac760a2f3 not found"
Jul 27 10:31:36 hostxx dockerd[20333]: time="2021-07-27T10:31:36.198814238Z" level=warning msg="Failed to unmount 54a45a2bb6dfea5836a27ef335026cf6f8c0fd0acd77f22aabd58ce726ee43ff mount /var/lib/docker/zfs/graph/54a45a2bb6dfea5836a27ef335026cf6f8c0fd0acd77f22aabd58ce726ee43ff: invalid argument" storage-driver=zfs
Jul 27 10:31:36 hostxx dockerd[20333]: time="2021-07-27T10:31:36.200495574Z" level=error msg="7ec3e2fbb1b340d989d92b8240d99891caa1119acdeb93227a643bbb7fcde52e cleanup: failed to delete container from containerd: no such container"
Jul 27 10:31:36 hostxx dockerd[20333]: time="2021-07-27T10:31:36.200644595Z" level=error msg="Handler for POST /v1.41/containers/7ec3e2fbb1b3/start returned error: error creating zfs mount: mount rpool/var/lib/docker/54a45a2bb6dfea5836a27ef335026cf6f8c0fd0acd77f22aabd58ce726ee43ff:/var/lib/docker/zfs/graph/54a45a2bb6dfea5836a27ef335026cf6f8c0fd0acd77f22aabd58ce726ee43ff: no such file or directory"
Jul 27 10:35:46 hostxx dockerd[20333]: time="2021-07-27T10:35:46.917181825Z" level=error msg="Error removing mounted layer 7ec3e2fbb1b340d989d92b8240d99891caa1119acdeb93227a643bbb7fcde52e: exit status 1: \"/sbin/zfs fs destroy -r rpool/var/lib/docker/54a45a2bb6dfea5836a27ef335026cf6f8c0fd0acd77f22aabd58ce726ee43ff\" => cannot open 'rpool/var/lib/docker/54a45a2bb6dfea5836a27ef335026cf6f8c0fd0acd77f22aabd58ce726ee43ff': dataset does not exist\n"
Jul 27 10:35:46 hostxx dockerd[20333]: time="2021-07-27T10:35:46.917516035Z" level=error msg="Handler for DELETE /v1.41/containers/7ec3e2fbb1b3 returned error: container 7ec3e2fbb1b340d989d92b8240d99891caa1119acdeb93227a643bbb7fcde52e: driver \"zfs\" failed to remove root filesystem: exit status 1: \"/sbin/zfs fs destroy -r rpool/var/lib/docker/54a45a2bb6dfea5836a27ef335026cf6f8c0fd0acd77f22aabd58ce726ee43ff\" => cannot open 'rpool/var/lib/docker/54a45a2bb6dfea5836a27ef335026cf6f8c0fd0acd77f22aabd58ce726ee43ff': dataset does not exist\n"
例如上面挂载时会将文件系统挂载到/var/lib/docker/zfs/graph/54a45a2bb6dfea5836a27ef335026cf6f8c0fd0acd77f22aabd58ce726ee43ff目录。
mount rpool/var/lib/docker/54a45a2bb6dfea5836a27ef335026cf6f8c0fd0acd77f22aabd58ce726ee43ff /var/lib/docker/zfs/graph/54a45a2bb6dfea5836a27ef335026cf6f8c0fd0acd77f22aabd58ce726ee43ff
通过查看官方文档发现：
Each running container’s unified filesystem is mounted on a mount point in /var/lib/docker/zfs/graph/
zfs.fsname
Set zfs filesystem under which docker will create its own datasets. By default docker will pick up the zfs filesystem where docker graph (/var/lib/docker) is located.
The base layer of an image is a ZFS filesystem. Each child layer is a ZFS clone based on a ZFS snapshot of the layer below it. A container is a ZFS clone based on a ZFS Snapshot of the top layer of the image it’s created from.
zfs_zpool
When you start a container, the following steps happen in order:
The base layer of the image exists on the Docker host as a ZFS filesystem.
Additional image layers are clones of the dataset hosting the image layer directly below it.
In the diagram, “Layer 1” is added by taking a ZFS snapshot of the base layer and then creating a clone from that snapshot. The clone is writable and consumes space on-demand from the zpool. The snapshot is read-only, maintaining the base layer as an immutable object.
- When the container is launched, a writable layer is added above the image.
In the diagram, the container’s read-write layer is created by making a snapshot of the top layer of the image (Layer 1) and creating a clone from that snapshot.
- As the container modifies the contents of its writable layer, space is allocated for the blocks that are changed. By default, these blocks are 128k.
通过查询docker container，docker image的dataset发现其dataset都指向rpool/var/lib/docker，如下所示
devops@edge01:~$ sudo docker info
Client:
Context: default
Debug Mode: false
Server:
Containers: 3
Running: 0
Paused: 0
Stopped: 3
Images: 3
Server Version: 20.10.2
Storage Driver: zfs
Zpool: rpool
Zpool Health: ONLINE
Parent Dataset: rpool/var/lib/docker // <== parent dataset
Space Used By Parent: 638941680
Space Available: 649245630688
Parent Quota: no
Compression: lz4
Logging Driver: json-file
devops@edge01:/var/lib/docker$ sudo docker inspect 7ec3e2fbb1b3
[
{
"Id": "7ec3e2fbb1b340d989d92b8240d99891caa1119acdeb93227a643bbb7fcde52e",
"Created": "2021-07-23T07:47:27.096840332Z",
"Path": "/bin/bash",
"Args": [
"-c",
"java -jar ${ENV_JAR_FILE_NAME}"
],
"State": {
"Status": "dead",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": true,
"Pid": 0,
"ExitCode": 255,
"Error": "container 7ec3e2fbb1b340d989d92b8240d99891caa1119acdeb93227a643bbb7fcde52e: driver \"zfs\" failed to remove root filesystem: exit status 1: \"/sbin/zfs fs destroy -r rpool/var/lib/docker/54a45a2bb6dfea5836a27ef335026cf6f8c0fd0acd77f22aabd58ce726ee43ff\" => cannot open 'rpool/var/lib/docker/54a45a2bb6dfea5836a27ef335026cf6f8c0fd0acd77f22aabd58ce726ee43ff': dataset does not exist\n",
"StartedAt": "2021-07-23T07:47:35.166881868Z",
"FinishedAt": "2021-07-27T09:53:46.508777404Z"
},
"Image": "sha256:8fb94d0ffb0cd47f20f8e94da17f463624c86a7417c9dc46f857b9a7f71cc4a9",
........
"GraphDriver": {
"Data": {
"Dataset": "rpool/var/lib/docker/54a45a2bb6dfea5836a27ef335026cf6f8c0fd0acd77f22aabd58ce726ee43ff", // <== dataset
"Mountpoint": "/var/lib/docker/zfs/graph/54a45a2bb6dfea5836a27ef335026cf6f8c0fd0acd77f22aabd58ce726ee43ff"
},
"Name": "zfs"
},
devops@edge01:~$ sudo docker image inspect 8fb94d0ffb0c
[
{
"Id": "sha256:8fb94d0ffb0cd47f20f8e94da17f463624c86a7417c9dc46f857b9a7f71cc4a9",
"RepoTags": [
"nexus.xxx/docker-repo/dataocean/dataocean-spark-agent:1.5"
],
"RepoDigests": [
"nexus.xxx/docker-repo/dataocean/dataocean-spark-agent@sha256:28a92f16b6dcc3e1d830db32f996c70bdb2ec7753c24bcfabcc6d9342ea434c7"
],
"Parent": "",
"Comment": "",
"Created": "2021-07-19T06:03:29.51948376Z",
"Container": "383c6a7dccfc1df3655e8a3f361c1fb89c4b2dbae8f9d40ef34f7dfbe28355f3",
.......
"Architecture": "amd64",
"Os": "linux",
"Size": 663244813,
"VirtualSize": 663244813,
"GraphDriver": {
"Data": {
"Dataset": "rpool/var/lib/docker/cb0300bea441e30808f29774cc90fde08197130f32a10b48412f1becfd9ae22d", // <== dataset
"Mountpoint": "/var/lib/docker/zfs/graph/cb0300bea441e30808f29774cc90fde08197130f32a10b48412f1becfd9ae22d"
},
"Name": "zfs"
},
"RootFS": {
"Type": "layers",
"Layers": [
"sha256:91d23cf5425acce8ca4402d83139cec63a5547d1a704af63478df008f9b2e4e0",
"sha256:f36b28e4310dbc9a2145292188f06c42b8464773b2ef21bd24f958c653555cd1",
"sha256:6cb741cb00b79a4ddfc008982d10db40b0b693740f1d8ce9eba6d18a0501cd0d",
"sha256:77008e118980159bff059c84efadeadd550767db65579466d2d0ad801577ff5d",
"sha256:d7f86666d1bcab3a642a16384175a92d0030d61257c1b9779f5785001590b45a",
"sha256:f7cd1e6d8bd1eccdb6bf671b53b9924da91f2aa8130c1cf117509651140446be",
"sha256:dd8fb3d1bab825dc7e59a25a868cafe4de0f1184a7ae7bc2e4b406a841b89629",
"sha256:ac8c46d60f6cab3c20b5653ad6e733cc8ab9fea2c628cb4b8a96cd96bfaa4c74",
"sha256:9e889161eed991eb83184c31a0c1a8db36afacbe642d927578ee9155d18ae062",
"sha256:1a8b1b4483dd175b73c32a24ca1bc1061fbc2edd6976d5e4e4cd7795e3c2d9a2",
"sha256:44b140dcb70f684adb382bbeaa42f691456c6091f123a4840195f398546fcd47",
"sha256:3b6cd15e63d8c22053f7531d3b7ea6cc7a56cdd9e4ede9ddc8a6bb962550da7d",
"sha256:d77587ac1b6a0823239b780d79cb69b09b85ca6d5d10fbfba7eb6c799534bcc2",
"sha256:9dfca223a0431c423ef669ac02b60eb5ddbc0ed94d815e89014abefa76f49790",
"sha256:58bd033a971a51c82b2abe5b161c523df3c9c598942cc93410b61b943b8d292d"
]
},
"Metadata": {
"LastTagTime": "0001-01-01T00:00:00Z"
}
}
]
但是发现docker container对应的dataset中的UUID都位于rpool/ROOT/ubuntu下，如下所示：
devops@edge01:~$ sudo zfs list | grep 54a45a2bb6dfea5836a27ef335026cf6f
rpool/ROOT/ubuntu/54a45a2bb6dfea5836a27ef335026cf6f8c0fd0acd77f22aabd58ce726ee43ff 146K 605G 219K legacy
rpool/ROOT/ubuntu/54a45a2bb6dfea5836a27ef335026cf6f8c0fd0acd77f22aabd58ce726ee43ff-init 384K 605G 473M legacy
devops@edge01:~$ sudo zfs list -d 3 rpool
NAME USED AVAIL REFER MOUNTPOINT
rpool 204G 605G 219K /
rpool/ROOT 3.55G 605G 219K none
rpool/ROOT/ubuntu 3.55G 605G 2.30G /
rpool/ROOT/ubuntu/023d0eadcaf6d09a16ced01cdd26fb3b23d8456dee9f148dd01369d2dc75c855 122M 605G 473M legacy
rpool/ROOT/ubuntu/0c38c9ff47d0a640ee4f440f077fc8177e1bf9ac23bd6d5c3529159e411403a1 503K 605G 93.1M legacy
rpool/ROOT/ubuntu/0d3f78484d56c6217d54fe1081c2d2b687db54150db9e34aea9a90c429101d72 292K 605G 352M legacy
rpool/ROOT/ubuntu/0e58b394e1bad6cfd1412485e7ace1d1a51cc15d98a9785a4aceb93639771edf 567K 605G 352M legacy
rpool/ROOT/ubuntu/0f319015f3c6b8c9c30cb33c3441882c78559d23fe1b3800d235c8b6f53640bd 292K 605G 352M legacy
rpool/ROOT/ubuntu/1a91a5c4ab766b40abeec21e56fbeb2ad8f5fabcb3df79aca911d37bb8d9b31a 256K 605G 352M legacy
rpool/ROOT/ubuntu/2ab9c1f6ef5e6de094f7109c5b5b40b6267caaf62344373145ce7c2e59c5125f 256K 605G 352M legacy
rpool/ROOT/ubuntu/35184f9d395e3807e0cf2ef49d800ddda3f5bbb44230cb16d3892560bcc21d42 548K 605G 352M legacy
rpool/ROOT/ubuntu/39147b611d63e9f769b98d3802e64bccef5800b5d4003e220e2d1ab7c7d5d470 256K 605G 93.1M legacy
rpool/ROOT/ubuntu/3de5b83ac71f6f903e51dfa65a8ea53a3160dfc9dd66a9fee5955d077abb761e 121M 605G 473M legacy
rpool/ROOT/ubuntu/410532bae085f0fd92b4f36d4d57b75ef9b9f9e60a7037cf454418e5075fe51f 512K 605G 352M legacy
rpool/ROOT/ubuntu/4ac7bc5cce120db782c61d51f712b79fdeb4a697cd51675afc0fd42d48b779b8 260M 605G 352M legacy
rpool/ROOT/ubuntu/51bc16024295a4d9115396fc05a2aeb29ae1f973f473f9e52695e1cda99073d2 256K 605G 352M legacy
rpool/ROOT/ubuntu/5264796623c36d053ffe86c05ce7933f4422fc3b6830e7aac1834d67105c905b 238K 605G 352M legacy
rpool/ROOT/ubuntu/54a45a2bb6dfea5836a27ef335026cf6f8c0fd0acd77f22aabd58ce726ee43ff 146K 605G 219K legacy
rpool/ROOT/ubuntu/54a45a2bb6dfea5836a27ef335026cf6f8c0fd0acd77f22aabd58ce726ee43ff-init 384K 605G 473M legacy
rpool/ROOT/ubuntu/5a7d6248cf15c5031fb4542d589fd482cf17f18f1e89abb6fe239e3e83f67a42 274K 605G 352M legacy
rpool/ROOT/ubuntu/619c7859d73c11b7b12e99201c7f84cef1fd398d4ae42187eb8213a34b8847ff 219K 605G 352M legacy
rpool/ROOT/ubuntu/7830ce34a980a3295295dd1810051e25559ce47906a58d18d9029275c33a2d7e 274K 605G 352M legacy
rpool/ROOT/ubuntu/91be5ca48fabb70811c8c52b968081855d2d7ac26a7222e21834e7d923938e8d 104M 605G 104M legacy
rpool/ROOT/ubuntu/94cb1098f614658651addf41c1cf710d2addcff88dee93deb4d5c562df1518d7 274K 605G 352M legacy
rpool/ROOT/ubuntu/9a49b72f22292e0c657a313e4c0eddcbaa49c158cf3eac5f19a26b225af24b86 256K 605G 352M legacy
rpool/ROOT/ubuntu/9ad4de619717f66dc18f13cb539f60c99c90353c07e3a1738d4434c7c83428d0 329K 605G 352M legacy
rpool/ROOT/ubuntu/a1482edf682b077088e4a185319a29ec05480ecb6a2a4a15090770f61ab71a24 274K 605G 352M legacy
rpool/ROOT/ubuntu/a20330135e8f29b7e3629ca2d7b38de2d0c89d320ab0ef315364f777825f9461 1.25M 605G 473M legacy
rpool/ROOT/ubuntu/a20330135e8f29b7e3629ca2d7b38de2d0c89d320ab0ef315364f777825f9461-init 384K 605G 473M legacy
rpool/ROOT/ubuntu/ad08a57e369e3e7e306f62875ca193d41ea3663039e0e1eb302f6758f681a2db 292K 605G 352M legacy
rpool/ROOT/ubuntu/b12cf1c1730a5fb39e028849e8f9f75fb0f581c0f45432dbc15fb4a3788b1463 402K 605G 93.1M legacy
rpool/ROOT/ubuntu/b6ec7440f38c89abd59b3b2330d6cbe1949b2acf274ead4cb718656c59d6baf8 292K 605G 352M legacy
rpool/ROOT/ubuntu/b7e7ab11564f2a966533a311e26c7de4c69bead11080002a5679419f05b09fbb 347K 605G 352M legacy
rpool/ROOT/ubuntu/cb0300bea441e30808f29774cc90fde08197130f32a10b48412f1becfd9ae22d 122M 605G 473M legacy
rpool/ROOT/ubuntu/cfdf5b8735a6f8dead97d458127e836645f88cbc92007d1f08d46299d83364ec 2.64M 605G 475M legacy
rpool/ROOT/ubuntu/cfdf5b8735a6f8dead97d458127e836645f88cbc92007d1f08d46299d83364ec-init 420K 605G 473M legacy
rpool/ROOT/ubuntu/d64614180b337cdf652a122390dd1485f81ad47f4aa10afb390835dc725c8a79 256K 605G 352M legacy
rpool/ROOT/ubuntu/e41df996003622f0c9da1558e5a0d347d9bfb98c0938e612836f5f9c2c2a1dd5 292K 605G 93.1M legacy
rpool/ROOT/ubuntu/e839f3d2db93143f810bfd1bc566b2c2a5ca0349faa0ade7408fb2582bfee3e5 384K 605G 352M legacy
rpool/ROOT/ubuntu/f652eaaafcbd29a3232df89148201d89847cd912dc425f374e8be6cd9941bc80 274K 605G 352M legacy
rpool/ROOT/ubuntu/f923e66c8d89c46e4b5b12cc33f2dc6691cc007fa08f892c75a39f3458b166f9 292K 605G 352M legacy
rpool/ROOT/ubuntu/fdce2c0ba8f67c228d2ea87ac4dd881ef56072142a6822cca008bca6f0070015 256K 605G 352M legacy
rpool/home 157G 605G 78.3G /home
rpool/home/xxx 64.3G 605G 64.3G /home/xxx
rpool/home/root 14.3G 605G 14.3G /root
rpool/opt 411M 605G 411M /opt
rpool/srv 347K 605G 219K /srv
rpool/swap 21.3G 626G 402K -
rpool/tmp 15.6G 605G 15.6G legacy
rpool/usr 166M 605G 219K /usr
rpool/usr/local 166M 605G 166M /usr/local
rpool/var 5.49G 605G 219K /var
rpool/var/cache 437M 605G 384M /var/cache
rpool/var/games 347K 605G 219K /var/games
rpool/var/lib 610M 605G 219K /var/lib
rpool/var/lib/AccountsService 366K 605G 219K /var/lib/AccountsService
rpool/var/lib/docker 609M 605G 609M /var/lib/docker
rpool/var/lib/nfs 457K 605G 311K /var/lib/nfs
rpool/var/log 4.47G 605G 4.47G legacy
rpool/var/mail 347K 605G 219K /var/mail
rpool/var/snap 347K 605G 219K /var/snap
rpool/var/spool 466K 605G 283K legacy
rpool/var/tmp 512K 605G 292K legacy
rpool/var/www 347K 605G 219K /var/www
所以docker在运行时会去rpool/var/lib/docker下去查找对应uuid的dataset，这是就会因为not found而执行失败。
解决方案
将docker使用的dataset从rpool/ROOT/ubuntu重命名到rpool/var/lib/docker下即可。（只有位于同一个pool才能执行此操作）
sudo zfs rename "$dataset" "$newDataset"
脚本如下：
#!/bin/bash
# Move all ZFS child datasets to a new parent dataset
if [ ! $# -eq 2 ]; then
echo "Please supply a source and destination ZFS dataset as the first and second parameter"
echo "Example: ./move_zfs_child_datasets.bash rpool/ROOT/ubuntu rpool/var/lib/docker"
exit;
fi
currentDatasetPath=$1
newDatasetPath=$2
currentDatasets=$(sudo zfs list -H -t filesystem -d 1 -o name "$currentDatasetPath" | tail +2)
for dataset in $currentDatasets ; do
echo "***find $dataset"
newDataset="$newDatasetPath/${dataset##*/}"
echo "Move $dataset to $newDataset"
sudo zfs rename "$dataset" "$newDataset"
done
执行脚本后docker能正常运行，查询dataset如下所示
devops@edge01:~$ sudo zfs list -d 4 rpool
NAME USED AVAIL REFER MOUNTPOINT
rpool 204G 605G 219K /
rpool/ROOT 2.83G 605G 219K none
rpool/ROOT/ubuntu 2.83G 605G 2.30G /
rpool/home 157G 605G 78.3G /home
rpool/home/xxx 64.3G 605G 64.3G /home/xxx
rpool/home/root 14.3G 605G 14.3G /root
rpool/opt 411M 605G 411M /opt
rpool/srv 347K 605G 219K /srv
rpool/swap 21.3G 626G 7.12M -
rpool/tmp 15.6G 605G 15.6G legacy
rpool/usr 166M 605G 219K /usr
rpool/usr/local 166M 605G 166M /usr/local
rpool/var 6.23G 605G 219K /var
rpool/var/cache 437M 605G 384M /var/cache
rpool/var/games 347K 605G 219K /var/games
rpool/var/lib 1.32G 605G 219K /var/lib
rpool/var/lib/AccountsService 366K 605G 219K /var/lib/AccountsService
rpool/var/lib/docker 1.32G 605G 607M /var/lib/docker
rpool/var/lib/docker/023d0eadcaf6d09a16ced01cdd26fb3b23d8456dee9f148dd01369d2dc75c855 122M 605G 473M legacy
rpool/var/lib/docker/0aa5830cbcbdace555d00be6d884ed32c821c3be37bde21323407b6d9e7fcb6c 896K 605G 473M legacy
rpool/var/lib/docker/0aa5830cbcbdace555d00be6d884ed32c821c3be37bde21323407b6d9e7fcb6c-init 384K 605G 473M legacy
rpool/var/lib/docker/0c38c9ff47d0a640ee4f440f077fc8177e1bf9ac23bd6d5c3529159e411403a1 503K 605G 93.1M legacy
rpool/var/lib/docker/0d3f78484d56c6217d54fe1081c2d2b687db54150db9e34aea9a90c429101d72 292K 605G 352M legacy
rpool/var/lib/docker/0e58b394e1bad6cfd1412485e7ace1d1a51cc15d98a9785a4aceb93639771edf 567K 605G 352M legacy
rpool/var/lib/docker/0f319015f3c6b8c9c30cb33c3441882c78559d23fe1b3800d235c8b6f53640bd 292K 605G 352M legacy
rpool/var/lib/docker/1a91a5c4ab766b40abeec21e56fbeb2ad8f5fabcb3df79aca911d37bb8d9b31a 256K 605G 352M legacy
rpool/var/lib/docker/2ab9c1f6ef5e6de094f7109c5b5b40b6267caaf62344373145ce7c2e59c5125f 256K 605G 352M legacy
rpool/var/lib/docker/35184f9d395e3807e0cf2ef49d800ddda3f5bbb44230cb16d3892560bcc21d42 548K 605G 352M legacy
rpool/var/lib/docker/39147b611d63e9f769b98d3802e64bccef5800b5d4003e220e2d1ab7c7d5d470 256K 605G 93.1M legacy
rpool/var/lib/docker/3de5b83ac71f6f903e51dfa65a8ea53a3160dfc9dd66a9fee5955d077abb761e 121M 605G 473M legacy
rpool/var/lib/docker/410532bae085f0fd92b4f36d4d57b75ef9b9f9e60a7037cf454418e5075fe51f 512K 605G 352M legacy
rpool/var/lib/docker/4ac7bc5cce120db782c61d51f712b79fdeb4a697cd51675afc0fd42d48b779b8 260M 605G 352M legacy
rpool/var/lib/docker/51bc16024295a4d9115396fc05a2aeb29ae1f973f473f9e52695e1cda99073d2 256K 605G 352M legacy
rpool/var/lib/docker/5264796623c36d053ffe86c05ce7933f4422fc3b6830e7aac1834d67105c905b 238K 605G 352M legacy
rpool/var/lib/docker/5a7d6248cf15c5031fb4542d589fd482cf17f18f1e89abb6fe239e3e83f67a42 274K 605G 352M legacy
rpool/var/lib/docker/619c7859d73c11b7b12e99201c7f84cef1fd398d4ae42187eb8213a34b8847ff 219K 605G 352M legacy
rpool/var/lib/docker/7830ce34a980a3295295dd1810051e25559ce47906a58d18d9029275c33a2d7e 274K 605G 352M legacy
rpool/var/lib/docker/91be5ca48fabb70811c8c52b968081855d2d7ac26a7222e21834e7d923938e8d 104M 605G 104M legacy
rpool/var/lib/docker/94cb1098f614658651addf41c1cf710d2addcff88dee93deb4d5c562df1518d7 274K 605G 352M legacy
rpool/var/lib/docker/9a49b72f22292e0c657a313e4c0eddcbaa49c158cf3eac5f19a26b225af24b86 256K 605G 352M legacy
rpool/var/lib/docker/9ad4de619717f66dc18f13cb539f60c99c90353c07e3a1738d4434c7c83428d0 329K 605G 352M legacy
rpool/var/lib/docker/a1482edf682b077088e4a185319a29ec05480ecb6a2a4a15090770f61ab71a24 274K 605G 352M legacy
rpool/var/lib/docker/a20330135e8f29b7e3629ca2d7b38de2d0c89d320ab0ef315364f777825f9461 1.25M 605G 473M legacy
rpool/var/lib/docker/a20330135e8f29b7e3629ca2d7b38de2d0c89d320ab0ef315364f777825f9461-init 384K 605G 473M legacy
rpool/var/lib/docker/ad08a57e369e3e7e306f62875ca193d41ea3663039e0e1eb302f6758f681a2db 292K 605G 352M legacy
rpool/var/lib/docker/b12cf1c1730a5fb39e028849e8f9f75fb0f581c0f45432dbc15fb4a3788b1463 402K 605G 93.1M legacy
rpool/var/lib/docker/b6ec7440f38c89abd59b3b2330d6cbe1949b2acf274ead4cb718656c59d6baf8 292K 605G 352M legacy
rpool/var/lib/docker/b7e7ab11564f2a966533a311e26c7de4c69bead11080002a5679419f05b09fbb 347K 605G 352M legacy
rpool/var/lib/docker/cb0300bea441e30808f29774cc90fde08197130f32a10b48412f1becfd9ae22d 122M 605G 473M legacy
rpool/var/lib/docker/cfdf5b8735a6f8dead97d458127e836645f88cbc92007d1f08d46299d83364ec 2.41M 605G 474M legacy
rpool/var/lib/docker/cfdf5b8735a6f8dead97d458127e836645f88cbc92007d1f08d46299d83364ec-init 420K 605G 473M legacy
rpool/var/lib/docker/d64614180b337cdf652a122390dd1485f81ad47f4aa10afb390835dc725c8a79 256K 605G 352M legacy
rpool/var/lib/docker/e41df996003622f0c9da1558e5a0d347d9bfb98c0938e612836f5f9c2c2a1dd5 292K 605G 93.1M legacy
rpool/var/lib/docker/e839f3d2db93143f810bfd1bc566b2c2a5ca0349faa0ade7408fb2582bfee3e5 384K 605G 352M legacy
rpool/var/lib/docker/f652eaaafcbd29a3232df89148201d89847cd912dc425f374e8be6cd9941bc80 274K 605G 352M legacy
rpool/var/lib/docker/f923e66c8d89c46e4b5b12cc33f2dc6691cc007fa08f892c75a39f3458b166f9 292K 605G 352M legacy
rpool/var/lib/docker/fdce2c0ba8f67c228d2ea87ac4dd881ef56072142a6822cca008bca6f0070015 256K 605G 352M legacy
rpool/var/lib/nfs 457K 605G 311K /var/lib/nfs
rpool/var/log 4.48G 605G 4.48G legacy
rpool/var/mail 347K 605G 219K /var/mail
rpool/var/snap 347K 605G 219K /var/snap
rpool/var/spool 466K 605G 283K legacy
rpool/var/tmp 512K 605G 292K legacy
rpool/var/www 347K 605G 219K /var/www
网友评论