最近开发环境跑的docker-compose应用隔一天会出现一次故障，导致docker无法运行，初步查看为/run目录占满导致
# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 986M 0 986M 0% /dev
tmpfs 997M 997M 89K 100% /dev/shm
tmpfs 997M 608K 997M 1% /run
tmpfs 997M 0 997M 0% /sys/fs/cgroup
/dev/vda1 40G 11G 27G 30% /
tmpfs 200M 0 200M 0% /run/user/0
docker命令无法使用，直接报一大堆错误￥%……&*,看不懂0.0
偶然一次在/run目录占到84%时，感觉docker应用出错了，迅速查看日志，
# docker-compse logs
mysql_1 | 2018-11-14T00:44:30.458672463Z 2018-11-14T00:38:31.363104Z 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 4799ms. The settings might not be optimal. (flushed=0 and evicted=0, during the time.)
mysql_1 | 2018-11-14T00:44:30.458676907Z 2018-11-14T00:39:02.566011Z 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 4398ms. The settings might not be optimal. (flushed=0 and evicted=0, during the time.)
mysql_1 | 2018-11-14T00:44:30.458680746Z 2018-11-14T00:40:20.468074Z 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 4507ms. The settings might not be optimal. (flushed=0 and evicted=0, during the time.)
mysql_1 | 2018-11-14T00:44:30.458684290Z 2018-11-14T00:42:00.666939Z 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 7205ms. The settings might not be optimal. (flushed=0 and evicted=0, during the time.)
mysql_1 | 2018-11-14T00:44:30.458687982Z 2018-11-14T00:43:04.265507Z 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 4500ms. The settings might not be optimal. (flushed=0 and evicted=0, during the time.)
mysql_1 | 2018-11-14T00:44:30.458691340Z 2018-11-14T00:44:26.565707Z 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 4701ms. The settings might not be optimal. (flushed=0 and evicted=0, during the time.)
都是mysql报错，随查资料，page_cleaner参数不合理导致
1、调整innodb_page_cleaners和innodb_buffer_pool_instances一致
2、调低innodb_lru_scan_depth=256，系统默认的1024大了点
或者
修改mysqld.cnf
under innodb_lock_wait_timeout = 50 add: innodb_flush_method=normal
restart MySql
mysql> show global variables like '%innodb_buffer_pool%';
+-------------------------------------+----------------+
| Variable_name | Value |
+-------------------------------------+----------------+
| innodb_buffer_pool_instances | 1 |
+-------------------------------------+----------------+
mysql> show global variables like '%innodb_page%';
+----------------------+-------+
| Variable_name | Value |
+----------------------+-------+
| innodb_page_cleaners | 1 |
| innodb_page_size | 16384 |
+--------------------- +-------+
mysql> show global variables like '%innodb_lru%';
+-----------------------+-------+
| Variable_name | Value |
+-----------------------+-------+
| innodb_lru_scan_depth | 1024 |
+-----------------------+-------+
# 修改为256
mysql> SET GLOBAL innodb_lru_scan_depth=256;
Query OK, 0 rows affected (0.00 sec)
修改完毕，重启mysql，但是又遇到问题了，
docker /bin/sh: fork: retry: No child processes
进程太多，无法创建新的进程
查看进程数和ulimit
# ps aux|wc -l
15765 # 具体数字记不清了
# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 7883
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 65535
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 7883
可以看到当前进程数时15765，大于max user processes（至于为什么是15765，个人认为是pending signals + max user processes，还需要再查资料来验证），很明显进程数超过ulimit限制，导致无法创建新的进程，那么kill掉进程docker命令应该就可以用了吧，但是到底是哪些进程过多导致的呢，联想到之前mysql的错误，mysql的嫌疑最大
# ps -ef|grep mysql|wc -l
15232 #具体数字记不清了
果然是mysql导致的，全部kill掉
# kill 1474 # mysql的父进程号
# ps aux|wc -l
38
重新修改mysqld.cnf，重启mysql
docker-compose restart mysql
观察几天，看问题是否最终解决
网友评论