[toc]

Windows in container

1 意义

可基于一个image run很多的container，每一个container都是独立的。所占用的存储空间仅仅是一个image的大小。并且基于容器技术，更方便迁移

2 架构

在Linux container（此处选用Ubuntu18.04）中安装kernel image，再安装kvm来运行虚拟机

image.png

3 前提

确保你的宿主机能够支持虚拟化

3.1 确认步骤

输入以下命令，如果输出大于0，则证明你的系统支持kvm虚拟化，如果输出等于0，则需要在BIOS中开启硬件虚拟化

sudo egrep -c '(vmx|svm)' /proc/cpuinfo

3.2 VMware虚拟机开启硬件虚拟化

VMware Workstations开启虚拟机虚拟化

image.png

VMware vCenter开启虚拟机虚拟化

image.png

4 步骤

4.1 安装docker

4.2 运行一个Ubuntu 18.04的容器

命令

docker run --privileged -p 3389:3389 -itd --name ubuntukvm --device=/dev/kvm --device=/dev/net/tun -v /sys/fs/cgroup:/sys/fs/cgroup:rw --cap-add=NET_ADMIN --cap-add=SYS_ADMIN ubuntu:18.04 /bin/bash

参数含义

--privileged：使用该参数能让container内的root拥有真正的root权限
--itd：it让容器开启一个伪终端，d让容器能够后台运行，避免出现退出容器终端后容器stop
--device：添加一个宿主机的设备到容器
-v：挂载目录
--cap-add：Add Linux capabilities

4.3 进入容器交互

命令

docker exec -it ubuntukvm /bin/bash

4.3.1 安装kvm工具

命令

apt-get update -y
apt-get install -y qemu-kvm libvirt-daemon-system libvirt-dev

4.3.2 服务启动

命令

chown root:kvm /dev/kvm
service libvirtd start
service virtlogd start

4.3.3 安装kernel image

命令

apt-get install -y linux-image-$(uname -r)

由于我的宿主机也是Ubuntu18.04，所以可以使用$(uname -r)，如果宿主机不是Ubuntu 18.04，需要去查找适合的kernel image版本

4.3.4 安装Vagrant

命令

apt-get install curl -y
apt-get install net-tools -y
apt-get install jq -y
vagrant_latest_version=$(curl -s https://checkpoint-api.hashicorp.com/v1/check/vagrant  | jq -r -M '.current_version')
echo $vagrant_latest_version
curl -O https://releases.hashicorp.com/vagrant/$(echo $vagrant_latest_version)/vagrant_$(echo $vagrant_latest_version)_x86_64.deb
dpkg -i vagrant_$(echo $vagrant_latest_version)_x86_64.deb
vagrant plugin install vagrant-libvirt

4.3.5 下载Windows 虚拟机Box

命令

mkdir /win10
cd /win10
vagrant init peru/windows-server-2012_r2-standard-x64-eval

4.3.6 启动虚拟机

命令及结果

root@15421e73344e:/win# VAGRANT_DEFAULT_PROVIDER=libvirt vagrant up
Bringing machine 'default' up with 'libvirt' provider...
==> default: Checking if box 'peru/windows-server-2012_r2-standard-x64-eval' version '20210104.01' is up to date...
==> default: Starting domain.
==> default: Error when updating domain settings: undefined method `strip' for nil:NilClass
==> default: Waiting for domain to get an IP address...
==> default: Waiting for SSH to become available...
==> default: Creating shared folders metadata...
==> default: Forwarding ports...
==> default: 3389 (guest) => 3389 (host) (adapter eth0)
==> default: 5986 (guest) => 5986 (host) (adapter eth0)
==> default: 5985 (guest) => 5985 (host) (adapter eth0)

4.3.7 网络配置

命令及结果

root@15421e73344e:/win# vagrant rdp
==> default: Detecting RDP info...
    default: Address: 192.168.121.46:3389
    default: Username: vagrant
==> default: Vagrant will now launch your RDP client with the connection parameters
==> default: above. If the connection fails, verify that the information above is
==> default: correct. Additionally, make sure the RDP server is configured and
==> default: running in the guest machine (it is disabled by default on Windows).
==> default: Also, verify that the firewall is open to allow RDP connections.
An appropriate RDP client was not found. Vagrant requires either
`xfreerdp` or `rdesktop` in order to connect via RDP to the Vagrant
environment. Please ensure one of these applications is installed and
available on the path and try again.

命令

iptables-save > $HOME/firewall.txt
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT

iptables -A FORWARD -i eth0 -o virbr1 -p tcp --syn --dport 3389 -m conntrack --ctstate NEW -j ACCEPT
iptables -A FORWARD -i eth0 -o virbr1 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -i virbr1 -o eth0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 3389 -j DNAT --to-destination 192.168.121.68
iptables -t nat -A POSTROUTING -o virbr1 -p tcp --dport 3389 -d 192.168.121.68 -j SNAT --to-source 192.168.121.1

iptables -D FORWARD -o virbr1 -j REJECT --reject-with icmp-port-unreachable
iptables -D FORWARD -i virbr1 -j REJECT --reject-with icmp-port-unreachable
iptables -D FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
iptables -D FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable

4.4 通过RDP连接虚拟机

4.4.1 局域网Windows连接

打开远程桌面连接

image.png

输入ip地址以及端口号进行连接

image.png

账号密码都为vagrant

image.png
image.png

5 创建docker image

5.1 commit 成新image

命令及结果

root@ubuntu:/# docker commit 15421e73344e  teym88/ubuntukvm
sha256:1e2f2b486882d835e77f790049ed7b805bae7871ef712ff8bf3f702ba71b4903

5.2 创建DockerFile

内容

FROM teym88/ubuntukvm
COPY startup.sh /

startup.sh内容

#!/bin/bash
#set -eou pipefail

chmod 777 /dev/kvm
service libvirtd start
service virtlogd start
cd win/
VAGRANT_DEFAULT_PROVIDER=libvirt vagrant up
vagrant rdp
iptables-save > $HOME/firewall.txt
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT

iptables -A FORWARD -i eth0 -o virbr1 -p tcp --syn --dport 3389 -m conntrack --ctstate NEW -j ACCEPT
iptables -A FORWARD -i eth0 -o virbr1 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -i virbr1 -o eth0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 3389 -j DNAT --to-destination 192.168.121.46
iptables -t nat -A POSTROUTING -o virbr1 -p tcp --dport 3389 -d 192.168.121.46 -j SNAT --to-source 192.168.121.1

iptables -D FORWARD -o virbr1 -j REJECT --reject-with icmp-port-unreachable
iptables -D FORWARD -i virbr1 -j REJECT --reject-with icmp-port-unreachable
iptables -D FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
iptables -D FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable

5.3 Build 新image

命令及结果

root@ubuntu:~/dockerwin# docker build -f Dockerfile --tag teym88/ubuntuwinkvm . 
Sending build context to Docker daemon  4.096kB
Step 1/2 : FROM teym88/ubuntukvm
 ---> 1e2f2b486882
Step 2/2 : COPY startup.sh /
 ---> d847dd970e49
Successfully built d847dd970e49
Successfully tagged teym88/ubuntuwinkvm:latest

5.4 使用新image运行

命令

root@ubuntu:~/dockerwin# docker run --privileged -itd --name kvmcontainer --device=/dev/kvm --device=/dev/net/tun -v /sys/fs/cgroup:/sys/fs/cgroup:rw --cap-add=NET_ADMIN --cap-add=SYS_ADMIN -p 3396:3389 teym88/ubuntuwinkvm /bin/bash
6abd67eb0c8f3e3202c04ad6179819712dfb666952b7f7115cf6637c88c8c439

5.5 到容器内部启动虚拟机

通过运行startup.sh即可

root@ubuntu:~/dockerwin# docker exec -it kvmcontainer /bin/bash
root@6abd67eb0c8f:/# ls
bin   dev  home        initrd.img.old  lib64  mnt  proc  run   srv         sys  usr                        var      vmlinuz.old
boot  etc  initrd.img  lib             media  opt  root  sbin  startup.sh  tmp  vagrant_2.2.14_x86_64.deb  vmlinuz  win
root@6abd67eb0c8f:/# st  
start-stop-daemon  stat               stdbuf             strings            strip              stty               
root@6abd67eb0c8f:/# /s                
sbin/       srv/        startup.sh  sys/        
root@6abd67eb0c8f:/# /startup.sh 
 * Starting libvirt management daemon libvirtd                                                                                                                             [ OK ] 
 * Starting libvirt logging daemon virtlogd                                                                                                                                [ OK ] 
Bringing machine 'default' up with 'libvirt' provider...
==> default: Checking if box 'peru/windows-server-2012_r2-standard-x64-eval' version '20210104.01' is up to date...
==> default: Starting domain.
==> default: Error when updating domain settings: undefined method `strip' for nil:NilClass
==> default: Waiting for domain to get an IP address...
==> default: Waiting for SSH to become available...
==> default: Creating shared folders metadata...
==> default: Forwarding ports...
==> default: 3389 (guest) => 3389 (host) (adapter eth0)
==> default: 5986 (guest) => 5986 (host) (adapter eth0)
==> default: 5985 (guest) => 5985 (host) (adapter eth0)
==> default: Machine already provisioned. Run `vagrant provision` or use the `--provision`
==> default: flag to force provisioning. Provisioners marked to run always will still run.
==> default: Detecting RDP info...
    default: Address: 192.168.121.46:3389
    default: Username: vagrant
==> default: Vagrant will now launch your RDP client with the connection parameters
==> default: above. If the connection fails, verify that the information above is
==> default: correct. Additionally, make sure the RDP server is configured and
==> default: running in the guest machine (it is disabled by default on Windows).
==> default: Also, verify that the firewall is open to allow RDP connections.
An appropriate RDP client was not found. Vagrant requires either
`xfreerdp` or `rdesktop` in order to connect via RDP to the Vagrant
environment. Please ensure one of these applications is installed and
available on the path and try again.

扩展

给虚拟机加盘

需要通过kvm方式进行添加

创建Block 类型的persistent volume

添加到Pod

yaml文件

将lstblk和blk10这两个pvc挂载到VmContainer的/dev/sdh和/dev/sdi

apiVersion: v1
kind: Pod
metadata:
  name: winvm
  labels: 
    app: containerVm
spec:
  nodeName: k8smaster
  containers:
  - name: win
    image: teym88/ubuntukvmwin
    command: ["/bin/bash", "-ce", "tail -f /dev/null"]
    securityContext:
      privileged : true
      capabilities:
        add: 
        - NET_ADMIN
        - SYS_ADMIN
    ports:
    - containerPort: 3389
    volumeMounts:
    - mountPath: /sys/fs/cgroup
      name: cgroup
    - mountPath: /sys
      name: sys
    volumeDevices:
    - name: data
      devicePath: /dev/sdh
    - name: blk10
      devicePath: /dev/sdi
  volumes:
  - name: cgroup
    hostPath:
      path: /sys/fs/cgroup
      type: Directory
  - name: sys
    hostPath:
      path: /sys
      type: Directory
  - name: data
    persistentVolumeClaim:
      claimName: lstblk
  - name: blk10
    persistentVolumeClaim:
      claimName: blk10 

注意

无法在线给Pod添加pvc，如果是使用pod来运行VmContainer，只能在一开始就添加pvc

The Pod "winvm" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)

尝试在vagrant up之前给虚拟机加盘

使用virsh编辑虚拟机xml文件

由于无法使用vagrant的方式来给虚拟机加盘，所以使用Libvirt的管理软件virsh来进行添加

查找虚拟机名称

进入pod 交互

root@k8smaster:~# kubectl exec -it winvm -- /bin/bash
root@winvm:/# ls

查看添加的硬盘

root@winvm:/# ls -l /dev |grep sd
brw-rw---- 1 root disk      8,    0 Mar 11 02:02 sda
brw-rw---- 1 root disk      8,    1 Mar 11 02:02 sda1
brw-rw---- 1 root disk      8,   16 Mar 11 02:02 sdb
brw-rw---- 1 root disk    147, 1003 Mar 11 02:02 sdh
brw-rw---- 1 root disk    147, 1004 Mar 11 02:02 sdi

查看虚拟机

root@winvm:/# virsh list --all      
 Id    Name                           State
----------------------------------------------------
 -     win_default                    shut off

编辑xml文件

使用virsh edit win_default命令进入虚拟机配置文件编辑模式，添加<disk>字段

<disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source dev='/dev/sdi'/>
      <target dev='vdb' bus='virtio'/>
</disk>

使用vagrant 命令开启虚拟机

root@winvm:/win# VAGRANT_DEFAULT_PROVIDER=libvirt vagrant up
Bringing machine 'default' up with 'libvirt' provider...
==> default: Checking if box 'peru/windows-server-2012_r2-standard-x64-eval' version '20210104.01' is up to date...
==> default: Starting domain.
==> default: Error when updating domain settings: undefined method `strip' for nil:NilClass
==> default: Waiting for domain to get an IP address...
==> default: Waiting for SSH to become available...
==> default: Creating shared folders metadata...
==> default: Forwarding ports...
==> default: 3389 (guest) => 3389 (host) (adapter eth0)
==> default: 5986 (guest) => 5986 (host) (adapter eth0)
==> default: 5985 (guest) => 5985 (host) (adapter eth0)
==> default: Machine already provisioned. Run `vagrant provision` or use the `--provision`
==> default: flag to force provisioning. Provisioners marked to run always will still run.

远程连接虚拟机

可以看到已经增加一个disk

image.png

尝试在虚拟机运行之后加盘

如上继续使用virsh编辑xml文件

使用vagrant reload 虚拟机

报错，无法重启

==> default: Clearing any previously set forwarded ports...
==> default: Halting domain...
Traceback (most recent call last):
        62: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/bin/vagrant:205:in `<main>'
        61: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/environment.rb:290:in `cli'
        60: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/cli.rb:67:in `execute'
        59: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/plugins/commands/reload/command.rb:40:in `execute'
        58: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/plugin/v2/command.rb:232:in `with_target_vms'
        57: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/plugin/v2/command.rb:232:in `each'
        56: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/plugin/v2/command.rb:243:in `block in with_target_vms'
        55: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/plugins/commands/reload/command.rb:42:in `block in execute'
        54: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/machine.rb:201:in `action'
        53: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/machine.rb:201:in `call'
        52: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/environment.rb:613:in `lock'
        51: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/machine.rb:215:in `block in action'
        50: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/machine.rb:246:in `action_raw'
        49: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/runner.rb:89:in `run'
        48: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/util/busy.rb:19:in `busy'
        47: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/runner.rb:89:in `block in run'
        46: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/builder.rb:149:in `call'
        45: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:48:in `call'
        44: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/builtin/call.rb:53:in `call'
        43: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/runner.rb:89:in `run'
        42: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/util/busy.rb:19:in `busy'
        41: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/runner.rb:89:in `block in run'
        40: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/builder.rb:149:in `call'
        39: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:48:in `call'
        38: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:127:in `block in finalize_action'
        37: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:48:in `call'
        36: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/builtin/config_validate.rb:25:in `call'
        35: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:48:in `call'
        34: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/builtin/config_validate.rb:25:in `call'
        33: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:48:in `call'
        32: from /root/.vagrant.d/gems/2.6.6/gems/vagrant-libvirt-0.3.0/lib/vagrant-libvirt/action/forward_ports.rb:199:in `call'
        31: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:48:in `call'
        30: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/builtin/call.rb:53:in `call'
        29: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/runner.rb:89:in `run'
        28: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/util/busy.rb:19:in `busy'
        27: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/runner.rb:89:in `block in run'
        26: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/builder.rb:149:in `call'
        25: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:48:in `call'
        24: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:127:in `block in finalize_action'
        23: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:48:in `call'
        22: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/builtin/call.rb:53:in `call'
        21: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/runner.rb:89:in `run'
        20: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/util/busy.rb:19:in `busy'
        19: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/runner.rb:89:in `block in run'
        18: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/builder.rb:149:in `call'
        17: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:48:in `call'
        16: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:127:in `block in finalize_action'
        15: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:48:in `call'
        14: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:127:in `block in finalize_action'
        13: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:48:in `call'
        12: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/builtin/call.rb:53:in `call'
        11: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/runner.rb:89:in `run'
        10: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/util/busy.rb:19:in `busy'
         9: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/runner.rb:89:in `block in run'
         8: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/builder.rb:149:in `call'
         7: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:48:in `call'
         6: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:127:in `block in finalize_action'
         5: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:48:in `call'
         4: from /root/.vagrant.d/gems/2.6.6/gems/vagrant-libvirt-0.3.0/lib/vagrant-libvirt/action/halt_domain.rb:28:in `call'
         3: from /root/.vagrant.d/gems/2.6.6/gems/fog-core-2.2.3/lib/fog/core/model.rb:72:in `wait_for'
         2: from /root/.vagrant.d/gems/2.6.6/gems/fog-core-2.2.3/lib/fog/core/wait_for.rb:6:in `wait_for'
         1: from /root/.vagrant.d/gems/2.6.6/gems/fog-core-2.2.3/lib/fog/core/wait_for.rb:6:in `loop'
/root/.vagrant.d/gems/2.6.6/gems/fog-core-2.2.3/lib/fog/core/wait_for.rb:9:in `block in wait_for': The specified wait_for timeout (30 seconds) was exceeded (Fog::Errors::TimeoutError)
        62: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/bin/vagrant:205:in `<main>'
        61: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/environment.rb:290:in `cli'
        60: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/cli.rb:67:in `execute'
        59: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/plugins/commands/reload/command.rb:40:in `execute'
        58: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/plugin/v2/command.rb:232:in `with_target_vms'
        57: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/plugin/v2/command.rb:232:in `each'
        56: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/plugin/v2/command.rb:243:in `block in with_target_vms'
        55: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/plugins/commands/reload/command.rb:42:in `block in execute'
        54: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/machine.rb:201:in `action'
        53: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/machine.rb:201:in `call'
        52: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/environment.rb:613:in `lock'
        51: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/machine.rb:215:in `block in action'
        50: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/machine.rb:246:in `action_raw'
        49: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/runner.rb:89:in `run'
        48: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/util/busy.rb:19:in `busy'
        47: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/runner.rb:89:in `block in run'
        46: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/builder.rb:149:in `call'
        45: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:48:in `call'
        44: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/builtin/call.rb:53:in `call'
        43: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/runner.rb:89:in `run'
        42: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/util/busy.rb:19:in `busy'
        41: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/runner.rb:89:in `block in run'
        40: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/builder.rb:149:in `call'
        39: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:48:in `call'
        38: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:127:in `block in finalize_action'
        37: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:48:in `call'
        36: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/builtin/config_validate.rb:25:in `call'
        35: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:48:in `call'
        34: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/builtin/config_validate.rb:25:in `call'
        33: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:48:in `call'
        32: from /root/.vagrant.d/gems/2.6.6/gems/vagrant-libvirt-0.3.0/lib/vagrant-libvirt/action/forward_ports.rb:199:in `call'
        31: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:48:in `call'
        30: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/builtin/call.rb:53:in `call'
        29: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/runner.rb:89:in `run'
        28: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/util/busy.rb:19:in `busy'
        27: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/runner.rb:89:in `block in run'
        26: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/builder.rb:149:in `call'
        25: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:48:in `call'
        24: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:127:in `block in finalize_action'
        23: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:48:in `call'
        22: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/builtin/call.rb:53:in `call'
        21: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/runner.rb:89:in `run'
        20: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/util/busy.rb:19:in `busy'
        19: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/runner.rb:89:in `block in run'
        18: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/builder.rb:149:in `call'
        17: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:48:in `call'
        16: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:127:in `block in finalize_action'
        15: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:48:in `call'
        14: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:127:in `block in finalize_action'
        13: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:48:in `call'
        12: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/builtin/call.rb:53:in `call'
        11: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/runner.rb:89:in `run'
        10: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/util/busy.rb:19:in `busy'
         9: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/runner.rb:89:in `block in run'
         8: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/builder.rb:149:in `call'
         7: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:48:in `call'
         6: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:127:in `block in finalize_action'
         5: from /opt/vagrant/embedded/gems/2.2.14/gems/vagrant-2.2.14/lib/vagrant/action/warden.rb:48:in `call'
         4: from /root/.vagrant.d/gems/2.6.6/gems/vagrant-libvirt-0.3.0/lib/vagrant-libvirt/action/halt_domain.rb:27:in `call'
         3: from /root/.vagrant.d/gems/2.6.6/gems/vagrant-libvirt-0.3.0/lib/vagrant-libvirt/action/halt_domain.rb:33:in `rescue in call'
         2: from /root/.vagrant.d/gems/2.6.6/gems/fog-libvirt-0.8.0/lib/fog/libvirt/models/compute/server.rb:107:in `poweroff'
         1: from /root/.vagrant.d/gems/2.6.6/gems/fog-libvirt-0.8.0/lib/fog/libvirt/requests/compute/vm_action.rb:7:in `vm_action'
/root/.vagrant.d/gems/2.6.6/gems/fog-libvirt-0.8.0/lib/fog/libvirt/requests/compute/vm_action.rb:7:in `destroy': Call to virDomainDestroyFlags failed: Failed to terminate process 274 with SIGKILL: Device or resource busy (Libvirt::Error)

使用virsh重启虚拟机

报错，一直处于shutting down的状态

root@winvm:/win# virsh destroy win_default
error: Disconnected from qemu:///system due to keepalive timeout
error: Failed to destroy domain win_default
error: internal error: connection closed due to keepalive timeout

root@winvm:/win# virsh list
 Id    Name                           State
----------------------------------------------------
 4     win_default                    in shutdown

root@winvm:/win# exit