Ovirt实现虚拟机通过NAT上网

作者: strongz | 来源:发表于2015-08-05 15:39 被阅读1774次

环境说明

OS: CentOS Linux release 7.1.1503 (Core)
Ovirt-engine: ovirt-engine-3.5.3.1-1.el7
VDSM: vdsm-4.16.20-0.el7
GuestOS: CentOS release 6.5 (Final)
硬件说明：单网卡且只有一个IP：10.10.19.100（可连接外网）
附：此主机同时充当engine和node角色

1. 安装Centos7-mini并update（省略）

2. 安装和配置ovirt

初始化相关系统设置

  [root@localhost ~]# echo "ovirthost01.ctcnet.com" >/etc/hostname
  
  [root@localhost ~]# systemctl stop NetworkManager
  [root@localhost ~]# systemctl disable NetworkManager

  [root@ovirthost01 ~]# cat >/etc/sysconfig/network-scripts/ifcfg-p4p1 <<EOF
  > DEVICE=p4p1
  > TYPE=Ethernet
  > ONBOOT=yes
  > BOOTPROTO=static
  > IPADDR=10.10.19.100
  > PFEFIX=24
  > GATEWAY=10.10.19.254
  > DNS1=10.10.19.254
  > EOF

  [root@ovirthost01 ~]# echo "10.10.19.100 ovirthost01 ovirthost01.ctcnet.com" >>/etc/hosts

  [root@ovirthost01 ~]# echo "export PATH=/bin:/sbin:$PATH" >>~/.bashrc &&source .bashrc &&echo $PATH

  [root@ovirthost01 ~]# setenforce 0 &&sed -i "s/SELINUX=enforcing/SELINUX=disabled/" /etc/selinux/config 
  
  [root@ovirthost01 ~]# systemctl stop firewalld && systemctl disable firewalld    
  
  [root@ovirthost01 ~]# systemctl stop iptables && systemctl disable iptables    
  
  [root@ovirthost01 ~]# yum install net-tools

  [root@ovirthost01 ~]# service postgresql initdb
  [root@ovirthost01 ~]# service postgresql start
  [root@ovirthost01 ~]# chkconfig postgresql on

安装和配置ovirt-engine

  [root@ovirthost01 ~]# yum localinstall -y http://resources.ovirt.org/pub/yum-repo/ovirt-release35.rpm 
  
  [root@ovirthost01 ~]# yum install -y ovirt-engine

  [root@ovirthost01 ~]# engine-setup

遇到如下错误：

  [ INFO  ] Generating post install configuration file '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'
  [ INFO  ] Stage: Transaction commit
  [ INFO  ] Stage: Closing up
  [ INFO  ] Restarting nfs services
  [ ERROR ] Failed to execute stage 'Closing up': Command '/bin/systemctl' failed to execute
  [ INFO  ] Stage: Clean up
            Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20150804004920-op260z.log
  [ INFO  ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20150804005258-setup.conf'
  [ INFO  ] Stage: Pre-termination
  [ INFO  ] Stage: Termination
  [ ERROR ] Execution of setup failed

检查日志：

  less /var/log/ovirt-engine/setup/ovirt-engine-setup-20150804004920-op260z.log

日志中记录了如下错误（启动nfs-server失败）：

  2015-08-04 00:52:58 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:932 execute-output: ('/bin/systemctl', 'start', 'nfs-server.service') stdout:

  2015-08-04 00:52:58 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:937 execute-output: ('/bin/systemctl', 'start', 'nfs-server.service') stderr:
  Job for nfs-server.service failed. See 'systemctl status nfs-server.service' and 'journalctl -xn' for details.
  
  2015-08-04 00:52:58 DEBUG otopi.context context._executeMethod:152 method exception
  Traceback (most recent call last):
    File "/usr/lib/python2.7/site-packages/otopi/context.py", line 142, in _executeMethod
      method['method']()
    File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/system/nfs.py", line 307, in _closeup
      state=state,
    File "/usr/share/otopi/plugins/otopi/services/systemd.py", line 138, in state
      'start' if state else 'stop'
    File "/usr/share/otopi/plugins/otopi/services/systemd.py", line 77, in _executeServiceCommand
      raiseOnError=raiseOnError
    File "/usr/lib/python2.7/site-packages/otopi/plugin.py", line 942, in execute
      command=args[0],
  RuntimeError: Command '/bin/systemctl' failed to execute

根据日志提示检查下nfs-server服务状态(failed)：

  [root@ovirthost01 ~]# systemctl status nfs-server.service

  nfs-server.service - NFS server and services
     Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled)
     Active: failed (Result: exit-code) since Tue 2015-08-04 00:52:58 EDT; 7min ago
    Process: 21922 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=1/FAILURE)
    Process: 21919 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
   Main PID: 21922 (code=exited, status=1/FAILURE)
     CGroup: /system.slice/nfs-server.service
  
  Aug 04 00:52:58 ovirthost01.ctcnet.com rpc.nfsd[21922]: rpc.nfsd: writing fd to kernel failed: errno 111 (Connection refused)
  Aug 04 00:52:58 ovirthost01.ctcnet.com rpc.nfsd[21922]: rpc.nfsd: unable to set any sockets for nfsd
  Aug 04 00:52:58 ovirthost01.ctcnet.com systemd[1]: nfs-server.service: main process exited, code=exited, status=1/FAILURE
  Aug 04 00:52:58 ovirthost01.ctcnet.com systemd[1]: Failed to start NFS server and services.
  Aug 04 00:52:58 ovirthost01.ctcnet.com systemd[1]: Unit nfs-server.service entered failed state.

搜索下得知要先启动rpcbind:

  [root@ovirthost01 ~]# systemctl start rpcbind

再次启动nfs-server并检查其状态(success)：

  [root@ovirthost01 ~]# systemctl start  nfs-server.service

  [root@ovirthost01 ~]# systemctl status nfs-server.service

  nfs-server.service - NFS server and services
     Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled)
     Active: active (exited) since Tue 2015-08-04 01:10:26 EDT; 11s ago
    Process: 22042 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS)
    Process: 22040 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
   Main PID: 22042 (code=exited, status=0/SUCCESS)
     CGroup: /system.slice/nfs-server.service
  
  Aug 04 01:10:26 ovirthost01.ctcnet.com systemd[1]: Starting NFS server and services...
  Aug 04 01:10:26 ovirthost01.ctcnet.com systemd[1]: Started NFS server and services.

再次运行engine-setup成功：

  [ INFO  ] Cleaning stale zombie tasks and commands
   
    --== CONFIGURATION PREVIEW ==--
   
    Firewall manager                        : firewalld
    Update Firewall                         : True
    Host FQDN                               : ovirthost01.ctcnet.com
    Engine database name                    : engine
    Engine database secured connection      : False
    Engine database host                    : localhost
    Engine database user name               : engine
    Engine database host name validation    : False
    Engine database port                    : 5432
    Engine installation                     : True
    PKI organization                        : ctcnet.com
    NFS mount point                         : /data/iso
    Configure WebSocket Proxy               : True
    Engine Host FQDN                        : ovirthost01.ctcnet.com
   
    Please confirm installation settings (OK, Cancel) [OK]: 
  [ INFO  ] Cleaning async tasks and compensations
  [ INFO  ] Checking the Engine database consistency
  [ INFO  ] Stage: Transaction setup
  [ INFO  ] Stopping engine service
  [ INFO  ] Stopping ovirt-fence-kdump-listener service
  [ INFO  ] Stopping websocket-proxy service
  [ INFO  ] Stage: Misc configuration
  [ INFO  ] Stage: Package installation
  [ INFO  ] Stage: Misc configuration
  [ INFO  ] Backing up database localhost:engine to '/var/lib/ovirt-engine/backups/engine-20150804012706.WAg6oL.dump'.
  [ INFO  ] Creating/refreshing Engine database schema
  [ INFO  ] Configuring WebSocket Proxy
  [ INFO  ] Generating post install configuration file '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'
  [ INFO  ] Stage: Transaction commit
  [ INFO  ] Stage: Closing up
           
            --== SUMMARY ==--
           
  [WARNING] Less than 16384MB of memory is available
            SSH fingerprint: F6:83:42:C9:FF:0F:A3:CE:ED:F5:85:EC:27:22:5F:E7
            Internal CA A3:CF:B5:C1:B4:29:8B:36:A5:9B:EB:69:99:A6:8D:5B:55:81:36:8F
            Web access is enabled at:
                http://ovirthost01.ctcnet.com:80/ovirt-engine
                https://ovirthost01.ctcnet.com:443/ovirt-engine
           
            --== END OF SUMMARY ==--
           
  [ INFO  ] Starting engine service
  [ INFO  ] Restarting httpd
  [ INFO  ] Stage: Clean up
            Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20150804012601-6zut1o.log
  [ INFO  ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20150804012750-setup.conf'
  [ INFO  ] Stage: Pre-termination
  [ INFO  ] Stage: Termination
  [ INFO  ] Execution of setup completed successfully

备份ovirt-engine文件和数据库

  [root@ovirthost01 ~]# engine-backup --mode=backup --file=engine_backup00 --log=./engine_backup00.log

如果想了解具体包含哪些配置文件，可以解压engine_bakcup00来查看：

  [root@ovirthost01 ~]# tar -jxvf engine_backup00

添加数据中心，集群，主机，数据存储域和ISO域（省略）

    Item              Name                 Path/IP
    -------------------------------------------------
    DataCenter        ctc_dc                    ——
    Cluster           ctc_cluster01            ——
    Host                host01                10.10.19.100
    Data_Stor_Dom     data_stor            10.10.19.99:/volume1/nfsstor
    ISO_Stor_Dom      iso_stor            10.10.19.100:/data/iso

注意：由于我采用vdsm和engine都在同一主机上，添加此主机到集群（安装vdsm）后会自动启动iptables,导致无法访问webadmin-portal（port:443/80）。添加相关条目到/etc/sysconfig/iptables来解决,内容如下：

  [root@ovirthost01 ~]# cat /etc/sysconfig/iptables

  # Generated by iptables-save v1.4.21 on Tue Aug  4 02:58:26 2015
  *filter
  :INPUT ACCEPT [0:0]
  :FORWARD ACCEPT [0:0]
  :OUTPUT ACCEPT [250:63392]
  -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
  -A INPUT -p icmp -j ACCEPT
  -A INPUT -i lo -j ACCEPT
  -A INPUT -p tcp -m tcp --dport 54321 -j ACCEPT
  -A INPUT -p tcp -m tcp --dport 111 -j ACCEPT
  -A INPUT -p udp -m udp --dport 111 -j ACCEPT
  -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
  -A INPUT -p udp -m udp --dport 161 -j ACCEPT
  -A INPUT -p tcp -m tcp --dport 16514 -j ACCEPT
  -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
  -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
  -A INPUT -p tcp -m multiport --dports 5900:6923 -j ACCEPT
  -A INPUT -p tcp -m multiport --dports 49152:49216 -j ACCEPT
  -A INPUT -j REJECT --reject-with icmp-host-prohibited
  -A FORWARD -m physdev ! --physdev-is-bridged -j REJECT --reject-with icmp-host-prohibited
  COMMIT
  # Completed on Tue Aug  4 02:58:26 2015

  [root@ovirthost01 ~]# systemctl restart iptables

3. 创建虚拟机（通过NAT连接外网）

1.导入iso镜像文件

    [root@ovirthost01 ~]# engine-iso-uploader -i iso_stor upload  /data/iso/*.iso

    Please provide the REST API password for the admin@internal oVirt Engine user (CTRL+D to abort): 
    Uploading, please wait...
    INFO: Start uploading /data/iso/CentOS-6.5-x86_64-minimal.iso 
    INFO: /data/iso/CentOS-6.5-x86_64-minimal.iso uploaded successfully
    INFO: Start uploading /data/iso/CentOS-7.0-1406-x86_64-Minimal.iso 
    INFO: /data/iso/CentOS-7.0-1406-x86_64-Minimal.iso uploaded successfully
    INFO: Start uploading /data/iso/virtio-win-0.1.96.iso 
    INFO: /data/iso/virtio-win-0.1.96.iso uploaded successfully

2.创建NAT网络配置文件/etc/libvirt/qemu/networks/nat.xml,内容如下

    <network>
        <name>nat</name>
        <uuid>b09d09a8-ebbd-476d-9045-e66012c9e83d</uuid>
        <forward mode='nat'/>
        <bridge name='natbr0' stp='on' delay='0' />
        <mac address='52:54:00:9D:82:DE'/>
        <ip address='192.168.1.1' netmask='255.255.255.0'>
            <dhcp>
                <range start='192.168.1.2' end='192.168.1.250' />
            </dhcp>
        </ip>
    </network>

3.通过libvirt/virsh创建NAT网络

    [root@ovirthost01 ~]# cat /etc/pki/vdsm/keys/libvirt_password 
    shibboleth
    
    [root@ovirthost01 ~]# virsh

    Welcome to virsh, the virtualization interactive terminal.
    
    Type:  'help' for help with commands
           'quit' to quit
    
    virsh # connect qemu:///system
    Please enter your authentication name: vdsm@ovirt
    Please enter your password: shibboleth
    
    virsh # net-list
     Name                 State      Autostart     Persistent
    ----------------------------------------------------------
     ;vdsmdummy;          active     no            no
     vdsm-ovirtmgmt       active     yes           yes
    
    virsh # net-define /etc/libvirt/qemu/networks/nat.xml
    Network nat defined from /etc/libvirt/qemu/networks/nat.xml
    
    virsh # net-autostart nat
    Network nat marked as autostarted
    
    virsh # net-start nat
    Network nat started
    
    virsh # net-list --all
     Name                 State      Autostart     Persistent
    ----------------------------------------------------------
     ;vdsmdummy;          active     no            no
     nat                  active     yes           yes
     vdsm-ovirtmgmt       active     yes           yes

    以上操作将创建nat功能的网桥，如下
    [root@ovirthost01 ~]# brctl show
    bridge name     bridge id               STP enabled     interfaces
    ;vdsmdummy;             8000.000000000000       no
    natbr0          8000.5254009d82de       yes             natbr0-nic
    ovirtmgmt               8000.b083fea27fed       no              p4p1

4.安装vdsm-hook-extnet

    [root@ovirthost01 ~]# yum install -y vdsm-hook-extnet

注：此处将下载extnet的hooks文件并存放到以下两目录

    [root@ovirthost01 ~]# ll /usr/libexec/vdsm/hooks/before_device_create
    total 4
    -rwxr-xr-x. 1 root root 1925 Jun  5 01:47 50_extnet
    [root@ovirthost01 ~]# ll /usr/libexec/vdsm/hooks/before_nic_hotplug
    total 4
    -rwxr-xr-x. 1 root root 1925 Jun  5 01:47 50_extnet

5.添加自定义设备属性extnet

    [root@ovirthost01 ~]# engine-config -s CustomDeviceProperties='{type=interface;prop={extnet=^[a-zA-Z0-9_ ---]+$}}'
    Please select a version:
    1. 3.0
    2. 3.1
    3. 3.2
    4. 3.3
    5. 3.4
    6. 3.5
    6

    [root@ovirthost01 ~]# engine-config -g CustomDeviceProperties
    CustomDeviceProperties:  version: 3.0
    CustomDeviceProperties:  version: 3.1
    CustomDeviceProperties:  version: 3.2
    CustomDeviceProperties:  version: 3.3
    CustomDeviceProperties: {type=interface;prop={SecurityGroups=^(?:(?:[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12}, *)*[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12}|)$}} version: 3.4
    CustomDeviceProperties: {type=interface;prop={extnet=^[a-zA-Z0-9_ ---]+$}} version: 3.5

    [root@ovirthost01 ~]# systemctl restart ovirt-engine

6.通过webadmin-portal创建虚拟机,并通过ISO安装GuestOS(省略)

7.添加nat端口配置集

图1-ovirt-nat0.JPG

8.添加vnic到虚拟机，并关联nat端口配置集

ovirt-nat1.JPG

9.进入到虚拟机验证（成功）

检查网卡是否添加

    [root@VM01-CentOS6 ~]# ifconfig -a

图3-ovirt-nat2.JPG

从vdsm主机的dhcp服务器处获取IP

    [root@VM01-CentOS6 ~]# dhclient eth0

图4-ovirt-nat3.JPG

通过ping外网来检查NAT是否成功

    [root@VM01-CentOS6 ~]# ping www.ovirt.org

图5-ovirt-nat4.JPG

10.参考资料

http://www.ovirt.org/VDSM-Hooks/network-nat
http://blog.lofyer.org/add-nat-ovirt-vdsm-hooks/
http://users.ovirt.narkive.com/WVp1moNk/ovirt-users-ovirt-3-5-nat
https://access.redhat.com/documentation/zh-CN/Red_Hat_Enterprise_Virtualization/3.5/html-single/Installation_Guide/index.html

网友评论

高机动老王:问个事，ovirt需要qemu-kvm支持么？
两者关系是啥
strongz:看下ovirt的架构就明白了：http://www.ovirt.org/documentation/architecture/architecture/
ok715:您好！NAT设置好了不能ping通host主机，只能上公网，请问怎么设置才能ping 通host主机呢？
ok715:@strongz 谢谢您！这个问题解决了！
strongz:@ok715 无法ping主机网的问题已解决，因为vdsm在系统启动是添加了策略路由/etc/sysconfig/network-scripts/rule-ovirtmgmt,route-ovirtmgmt。导致nat后再ping主机ip的包被丢弃，可以删除以上文件（rule,route）并同时删除ip rule和对应ip route来解决。也可以删除文件后重启主机来解决。
strongz:@ok715 这个应该是个bug,maillist里也有人提过。但没看到最终解决方法，可能3.6版有解决吧，没测过，你可以试试。

Ovirt实现虚拟机通过NAT上网

环境说明

1. 安装Centos7-mini并update（省略）

2. 安装和配置ovirt

3. 创建虚拟机（通过NAT连接外网）

相关文章

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读

虚拟化技术

Ovirt

@IT·互联网

程序员