[toc]

指令

usage: cinder extend <volume> <new_size>
Attempts to extend size of an existing volume.
Positional arguments:
  <volume>    Name or ID of volume to extend.
  <new_size>  New size of volume, in GiBs.

功能说明

最新的openstack(3.42)代码支持了热扩容功能：

Add ability to extend 'in-use' volume. User should be aware of the
whole environment before using this feature because it's dependent
on several external factors below:
1. nova-compute version - needs to be the latest for Pike.
2. only the libvirt compute driver supports this currently.
3. only iscsi and fibre channel volume types are supported
on the nova side currently.
Administrator can disable this ability by updating the
'volume:extend_attached_volume' policy rule. Extend in reserved
state is intentionally NOT allowed.

注意：

• 目前只有 libvirt 驱动能支持，且只有 iscsi 和 fibre channel 协议的卷类型能支持！
• cinder、nova做完热扩容后，fdisk -l查看可以看到磁盘大小变大，但是之前做的文件系统的大小是不会变化的，df -h 的结果跟扩容前一样。用户还必须用resize2fs 指令自行resize文件系统！

《openstack 官网对于热扩容在 CINDER 方面的说明》

《openstack 官网对于热扩容在 NOVA 方面的说明》

Cinder方面代码

Cinder 方面其实没有什么改动，跟冷扩容做的事一样，扩容后的size必须是整数GB且大于扩容前size。核心实现是调用卷驱动里的方法，如netapp，调用的是cinder.volume.drivers.netapp.dataontap.client.client_base.Client#do_direct_resize实现。唯一的区别是，扩容后，会调用nova api。

1. cinder.api.contrib.volume_actions.VolumeActionsController#_extend：

# 如果版本是3.42 且 卷状态是in-use，则走热扩容
if req_version.matches("3.42") and volume.status in ['in-use']:
    self.volume_api.extend_attached_volume(context, volume, size)
else: # 否则，冷扩展
    self.volume_api.extend(context, volume, size)

2. cinder/volume/manager.py里调用nova api，通知nova要扩展卷：

 if orig_volume_status == 'in-use':
        nova_api = compute.API()
        instance_uuids = [attachment.instance_uuid
                          for attachment in attachments]
        nova_api.extend_volume(context, instance_uuids, volume.id)

这里调用nova client 里的：
novaclient.v2.server_external_events.ServerExternalEventManager#create:

class ServerExternalEventManager(base.Manager):
    resource_class = Event

    def create(self, events):
        """Create one or more server events.

        :param:events: A list of dictionaries containing 'server_uuid', 'name',
                       'status', and 'tag' (which may be absent)
        """

        body = {'events': events}
        return self._create('/os-server-external-events', body, 'events',
                            return_raw=True)

'events'：

{'name': 'volume-extended',
'server_uuid': server_id,
'tag': volume_id}

Nova方面代码

Nova将使用现有的外部事件API端点监听来自Cinder的附加卷扩展通知。收到通知后，Nova将使用os-brick触发主机上的设备重新扫描，以发现卷大小的变化。

nova api '/os-server-external-events'：

• request：

    {
        "events": [
            {
                "name": "volume-extended",
                "server_uuid": "3df201cf-2451-44f2-8d25-a4ca826fc1f3",
                "tag": "0e63d806-6fe4-4ffc-99bf-f3dd056574c0"
            }
        ]
    }

• response：

    {
        "events": [
            {
                "name": "volume-extended",
                "status": "completed",
                "code": 200,
                "server_uuid": "3df201cf-2451-44f2-8d25-a4ca826fc1f3",
                "tag": "0e63d806-6fe4-4ffc-99bf-f3dd056574c0"
            }
        ]
    }

1. 检查nova驱动compute_driver = libvirt.LibvirtDriver是否能支持卷扩展 "supports_extend_volume": True
   驱动的支持功能定义在nova.virt.libvirt.driver.LibvirtDriver：

class LibvirtDriver(driver.ComputeDriver):
    capabilities = {
        "has_imagecache": True,
        "supports_recreate": True,
        "supports_migrate_to_same_host": False,
        "supports_attach_interface": True,
        "supports_device_tagging": True,
        "supports_tagged_attach_interface": True,
        "supports_tagged_attach_volume": True,
        "supports_extend_volume": True,
    }

2. 根据卷connection_info找对应的驱动，然后调用驱动的extend_volume方法。

    def _extend_volume(self, connection_info, instance):
        vol_driver = self._get_volume_driver(connection_info)
        return vol_driver.extend_volume(connection_info, instance)

3. 比如Iscsi，就会找nova.virt.libvirt.volume.iscsi.LibvirtISCSIVolumeDriver#extend_volume：

        new_size = self.connector.extend_volume(connection_info['data'])

4. 在调用ISCSIConnector的extend_volume
   os_brick.initiator.connectors.iscsi.ISCSIConnector#extend_volume：

        volume_paths = self.get_volume_paths(connection_properties)
        if volume_paths:
            return self._linuxscsi.extend_volume(volume_paths)

5. 调用linuxscsi的extend_volume
   os_brick.initiator.linuxscsi.LinuxSCSI#extend_volume:

   def extend_volume(self, volume_paths):
        """Signal the SCSI subsystem to test for volume resize.

        This function tries to signal the local system's kernel
        that an already attached volume might have been resized.
        """
        LOG.debug("extend volume %s", volume_paths)

        for volume_path in volume_paths:
            device = self.get_device_info(volume_path)
            LOG.debug("Volume device info = %s", device)
            device_id = ("%(host)s:%(channel)s:%(id)s:%(lun)s" %
                         {'host': device['host'],
                          'channel': device['channel'],
                          'id': device['id'],
                          'lun': device['lun']})
                          
            scsi_path = ("/sys/bus/scsi/drivers/sd/%(device_id)s" %
                         {'device_id': device_id})  
            # 如：scsi_path = u'/sys/bus/scsi/drivers/sd/3:0:0:3'

            # 获取设备大小
            size = self.get_device_size(volume_path)
            LOG.debug("Starting size: %s", size)

            # now issue the device rescan
            rescan_path = "%(scsi_path)s/rescan" % {'scsi_path': scsi_path}
            # 如：rescan_path = u'/sys/bus/scsi/drivers/sd/3:0:0:3/rescan'
            
            # 在rescan_path文件里写入1。
            # 对于SCSI设备，对 rescan 文件设置为1 可以使SCSI设备重新扫描。
            self.echo_scsi_command(rescan_path, "1")
            new_size = self.get_device_size(volume_path)
            LOG.debug("volume size after scsi device rescan %s", new_size)

        # 通过指令`/lib/udev/scsi_id --page 0x83 --whitelisted /dev/disk/by-path/ip-172.24.3.180:3260-iscsi-iqn.1992-08.com.netapp:sn.2d72abb030d511e7875800a098ac0ce9:vs.24-lun-3 `拿到wwnid
        scsi_wwn = self.get_scsi_wwn(volume_paths[0])
        # 如：scsi_wwn = u'3600a09803830387461244a62344f6b52'
        mpath_device = self.find_multipath_device_path(scsi_wwn)
        # 如果mpath_device非空，说明是多路径磁盘。
        if mpath_device:
            # Force a reconfigure so that resize works
            # 执行 `multipathd reconfigure` 重新获取 multipath.conf 配置中变化。
            self.multipath_reconfigure()
            # 获取设备大小
            size = self.get_device_size(mpath_device)
            LOG.info("mpath(%(device)s) current size %(size)s",
                     {'device': mpath_device, 'size': size})
                     
            # 调用指令 multipathd resize map multipath_device 重置多路径磁盘的大小
            result = self.multipath_resize_map(scsi_wwn)
            if 'fail' in result:
                LOG.error("Multipathd failed to update the size mapping of "
                          "multipath device %(scsi_wwn)s volume %(volume)s",
                          {'scsi_wwn': scsi_wwn, 'volume': volume_paths})
                return None
                
            # 获取设备大小
            new_size = self.get_device_size(mpath_device)
            LOG.info("mpath(%(device)s) new size %(size)s",
                     {'device': mpath_device, 'size': new_size})

        return new_size

流程简要：

1. 向scsi扫描文件写入 1：tee -a 1 "%(scsi_path)s/rescan"，对于SCSI设备，对 rescan 文件设置为1 可以使SCSI设备重新扫描。

2. /lib/udev/scsi_id --page 0x83 --whitelisted 得到scsi_wwn

3. 检查多路径设备，先检查/dev/disk/by-id/dm-uuid-mpath-%(wwn)s是否存在，存在则返回路径；如果不存在，则检查/dev/mapper/%(wwn)s，存在则返回路径，不存在返回none。

4. 如果第3部结果为none，流程结束。

5. 如果第3步查到路径，则需要执行 multipathd reconfigure 重新获取 multipath.conf 配置中变化。

6. 执行multipathd resize map scsi_wwn 重新设置设备大小。多路径磁盘流程结束。

FC和Iscsi驱动都是这样流程。

附：

centos 如何使用多路径磁盘

1.安装multipath工具

# 安装multipath工具
yum -y install device-mapper device-mapper-multipath

#加载multipath内核模块
modprobe dm-multipath
modprobe dm-round-robin

# 拷贝默认配置文件
cat /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf > /etc/multipath.conf

#启动服务
service multipathd start 或者 /bin/systemctl start  multipathd.service

2.配置nova_compute的配置文件

[libvirt]
#  iscsi磁盘是否开启多路径,True为开启
volume_use_multipath=True

# 建立连接时的rescan次数
num_iscsi_scan_tries=5

# 创建iscsi连接使用的iface名称
iscsi_iface=default

修改在线多路径设备容量的方法

redhat官网上介绍了修改在线多路径设备容量的方法：

RESIZING AN ONLINE MULTIPATH DEVICE

If you need to resize an online multipath device, use the following procedure.

1. Resize your physical device.
2. Execute the following command to find the paths to the LUN:

# multipath -l

3. Resize your paths. For SCSI devices, writing a 1 to the rescan file for the device causes the SCSI driver to rescan, as in the following command:

# echo 1 > /sys/block/path_device/device/rescan

Ensure that you run this command for each of the path devices. For example, if your path devices are sda, sdb, sde, and sdf, you would run the following commands:

# echo 1 > /sys/block/sda/device/rescan

# echo 1 > /sys/block/sdb/device/rescan

# echo 1 > /sys/block/sde/device/rescan

# echo 1 > /sys/block/sdf/device/rescan

4. Resize your multipath device by executing the multipathd resize command:

# multipathd resize map multipath_device

5. Resize the file system (assuming no LVM or DOS partitions are used):

# resize2fs /dev/mapper/mpatha

看的出，跟nova的处理流程差不多，主要少了resize2fs一步。

测试过程遇到问题

1. 问题： 创建虚机时提示"Host 'localhost.localdomain' is not mapped to any cell"

   解决： 执行 nova-manage cell_v2 simple_cell_setup

2. 问题： n-cpu日志提示libvirt 版本过低

   解决： 按照以下步骤升级
   $ rpm -qa|grep kvm
   $ rpm -qa|grep qemu-system
   $ sudo virsh -c qemu:///system version --daemon
   $ sudo yum remove qemu-system-x86
   $ sudo service libvirtd restart
   $ sudo virsh -c qemu:///system version --daemon

3. 问题： c-vol 提示：ERROR oslo_messaging.rpc.server ConnectionFailedError: HTTPConnectionPool(host='172.24.2.218', port=2379):
   Max retries exceeded with url: /v3alpha/lease/grant (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x718cc10>:
   Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
   ERROR oslo_messaging.rpc.server

   解决： 配错了coordination。在cinder.conf里吧[coordination]的backend_url注释掉即可。