Linux故障模拟

1. 模拟硬盘拔盘，插盘

(ENV) [root@ceph-1 ~]# lsscsi
[0:0:0:0]    disk    VMware   Virtual disk     1.0   /dev/sda
[0:0:1:0]    disk    VMware   Virtual disk     1.0   /dev/sdb
[0:0:2:0]    disk    VMware   Virtual disk     1.0   /dev/sdc
[1:0:0:0]    cd/dvd  NECVMWar VMware IDE CDR00 1.00  /dev/sr0

echo "scsi remove-single-device <2:0:0:1>" > /proc/scsi/scsi

echo "scsi add-single-device <2:0:0:1>" > /proc/scsi/scsi

echo 1> /sys/block/sdc/device/delete
echo "- - -"  > /sys/class/scsi_host/host2/scan

2. 模拟服务器掉电

echo b > /proc/sysrq-trigger

3. 模拟网络中断

ifconfig ip down/up

4. 修改coredump配置

echo "/var/crash/core-%e-%p-%t" > /proc/sys/kernel/core_pattern
ulimit -c unlimited

5. 修复yum db

rm -rf __db.00*
rpm --rebuilddb

5. TC Traffic Control 用来模拟网络环境

6. 升级内核版本

# 导入elrepo的key，然后安装elrepo的yum源
rpm -import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm

# 使用以下命令列出可用的内核相关包，如下图：
yum --disablerepo="*" --enablerepo="elrepo-kernel" list available

# 安装主线稳定内核
yum -y --enablerepo=elrepo-kernel install kernel-ml.x86_64 kernel-ml-devel.x86_64

# 查看内核版本默认启动顺序：
awk -F\' '$1=="menuentry " {print $2}' /etc/grub2.cfg
# 需要修改内核启动顺序为0

# 修改grub中默认版本启动顺序：
vim /etc/default/grub
将GRUB_DEFAULT=saved改为GRUB_0=saved，保存退出vim。

# 运行grub2-mkconfig命令来重新创建内核配置：
grub2-mkconfig -o /boot/grub2/grub.cfg

# 重启：reboot

# 待启动完毕，查看系统内核：uname -r

7. 刷新disk
   This method does not use any previous programs because it comunicates directly with the kernel interface. Because of that, there is no error recovering if something wrong happens while reloading your partition table. You are warned. I'm not responsible for any data loss or kryptonite damage you might suffer. :)
   As root user, send the following signal to the kernel interface:

# echo 1 > /sys/block/sdX/device/rescan (Where X is your device letter)

Then, check the dmesg reported messages to confirm that the kernel done the reloading of the device's partition table.
After this, your partition(s) should be recognized and enabled in /dev, making it possible to format, label and mount them!
Or:

rescan-scsi-bus.sh --remove --issue-lip

8. 查看指定systemd服务的log

journalctl -b -u <service>