美文网首页
“Cinder show 不存在的id,查询时间很长”分析

“Cinder show 不存在的id,查询时间很长”分析

作者: 笨手笨脚越 | 来源:发表于2017-12-12 16:12 被阅读20次

    问题: 使用“cinder show”查询一个不存在的volumeid,使用时间很长。

    [root@node1 ~]# time cinder show aksjdfkajsdkfjsakdf
    ERROR: No volume with a name or ID of 'aksjdfkajsdkfjsakdf' exists.
    real    0m35.374s
    user    0m0.855s
    sys     0m0.107s
    

    分析:

    1. cinderclient.v2.shell.do_show说明

    指令“cinder show”的入口函数是cinderclient.v2.shell.do_show

    @utils.arg('volume',
               metavar='<volume>',
               help='Name or ID of volume.')
    def do_show(cs, args):
        """Shows volume details."""
        info = dict()
        volume = utils.find_volume(cs, args.volume)
        info.update(volume._info)
    
        if 'readonly' in info['metadata']:
            info['readonly'] = info['metadata']['readonly']
    
    info.pop('links', None)
    # 打印字典
        utils.print_dict(info,
                         formatters=['metadata', 'volume_image_metadata',
                                     'attachments'])
    

    @utils.arg 是用于定义参数的标签。

    指令参数根据必填、选填分为两种:

    • 1)@utils.arg('volume', metavar='<volume>', help='Name or ID of volume.')
      必填参数,使用时不用带参数名。例如”cinder show 123123”,这里的123123就是<volume>。
    • 2)@utils.arg('--tenant', type=str, dest='tenant', nargs='?', metavar='<tenant>', help='Display information from single tenant (Admin only).')
      选填参数,名字多了个破折号 --,指令使用时必须带参数名。例如”cinder list --tenant 123123”

    根据参数值分类:

    • 1)@utils.arg('volume', metavar='<volume>', nargs='+', help='Name or ID of volume or volumes to delete.')
      如果设置nargs='+',代表值为数组型,以空格分隔,输入1个以上。如”cinder delete 111 2222 3333”,删除volume_id=111、volume_id=222、volume_id=333 的volume。
      此外还有nargs='*' 表示参数可设置零个或多个;nargs=' '+' 表示参数可设置一个或多个;nargs='?' 表示参数可设置零个或一个。
    • 2)@utils.arg('--metadata', type=str, nargs='*', metavar='<key=value>', default=None, help='Snapshot metadata key and value pairs. Default=None.')
      设置metavar='<key=value>',值为键值对形式。比如”cinder storage-create --metadata storage_protocol='fc' storage_ip='172.24.1.21' ”

    @utils.arg定义的参数会传入def do_show(cs, args)的args,通过args.xxxx来调用。

    2.utils.find_volume(cs, args.volume) 说明

    cinderclient.utils.find_volume:
    def find_volume(cs, volume):
        """Get a volume by name or ID."""
    return find_resource(cs.volumes, volume)
    def find_resource(manager, name_or_id):
        """Helper for the _find_* methods."""
        # first try to get entity as integer id
        try:
            if isinstance(name_or_id, int) or name_or_id.isdigit():
                return manager.get(int(name_or_id))
        except exceptions.NotFound:
            pass
        else:
            # now try to get entity as uuid
            try:
                uuid.UUID(name_or_id)
                return manager.get(name_or_id)
            except (ValueError, exceptions.NotFound):
                pass
        if sys.version_info <= (3, 0):
            name_or_id = encodeutils.safe_decode(name_or_id)
        try:
            try:
                resource = getattr(manager, 'resource_class', None)
                name_attr = resource.NAME_ATTR if resource else 'name'
                return manager.find(**{name_attr: name_or_id})
            except exceptions.NotFound:
                pass
            # finally try to find entity by human_id
            try:
                return manager.find(human_id=name_or_id)
            except exceptions.NotFound:
                msg = "No %s with a name or ID of '%s' exists." % \
                    (manager.resource_class.__name__.lower(), name_or_id)
                raise exceptions.CommandError(msg)
        except exceptions.NoUniqueMatch:
            msg = ("Multiple %s matches found for '%s', use an ID to be more"
                   " specific." % (manager.resource_class.__name__.lower(),
                                   name_or_id))
            raise exceptions.CommandError(msg)
    

    调用了cinderclient.utils.find_resource(manager, name_or_id)方法,find_resource的逻辑是:
    ① 如果id是int型,直接调用VolumeManager.get。如果没查到,走下一步。
    ② id转换成uuid,直接调用VolumeManager.get。如果没查到,走下一步。
    ③ 如果volumes类定义了NAME_ATTR属性,用{name_attr: name_or_id}作为条件调用VolumeManager.find查询。如果没查到,走下一步。
    ④ 用{human_id:name_or_id}作为条件调用VolumeManager.find查询。

    3.VolumeManager.get说明

    cinderclient.v2.volumes.VolumeManager#get
        def get(self, volume_id):
            """Get a volume.
            :param volume_id: The ID of the volume to get.
            :rtype: :class:`Volume`
            """
            return self._get("/volumes/%s" % volume_id, "volume")
    
    self._get调用cinderclient.base.Manager#_get
        def _get(self, url, response_key=None):
            resp, body = self.api.client.get(url)
            if response_key:
                return self.resource_class(self, body[response_key], loaded=True,
                                           resp=resp)
            else:
                return self.resource_class(self, body, loaded=True, resp=resp)
    

    self.resource_class在cinderclient.v2.volumes.VolumeManager定义:resource_class = Volume。所以返回值是body[response_key]转换出的cinderclient.v2.volumes.Volume(base.Resource)对象。

    4.VolumeManager.find说明

    VolumeManager.find调用cinderclient.base.ManagerWithFind#findall(self, **kwargs):

        def find(self, **kwargs):
            """
            Find a single item with attributes matching ``**kwargs``.
            This isn't very efficient for search options which require the
            Python side filtering(e.g. 'human_id')
            """
            matches = self.findall(**kwargs)
            num_matches = len(matches)
            if num_matches == 0:
                msg = "No %s matching %s." % (self.resource_class.__name__, kwargs)
                raise exceptions.NotFound(404, msg)
            elif num_matches > 1:
                raise exceptions.NoUniqueMatch
            else:
                matches[0].append_request_ids(matches.request_ids)
                return matches[0]
        def findall(self, **kwargs):
            """
            Find all items with attributes matching ``**kwargs``.
            This isn't very efficient for search options which require the
            Python side filtering(e.g. 'human_id')
            """
            # Want to search for all tenants here so that when attempting to delete
            # that a user like admin doesn't get a failure when trying to delete
            # another tenant's volume by name.
            search_opts = {'all_tenants': 1}
            # Pass 'name' or 'display_name' search_opts to server filtering to
            # increase search performance.
            if 'name' in kwargs:
                search_opts['name'] = kwargs['name']
            elif 'display_name' in kwargs:
                search_opts['display_name'] = kwargs['display_name']
            found = common_base.ListWithMeta([], None)
            searches = kwargs.items()
            listing = self.list(search_opts=search_opts)
            found.append_request_ids(listing.request_ids)
            # Not all resources attributes support filters on server side
            # (e.g. 'human_id' doesn't), so when doing findall some client
            # side filtering is still needed.
            for obj in listing:
                try:
                    if all(getattr(obj, attr) == value
                           for (attr, value) in searches):
                        found.append(obj)
                except AttributeError:
                    continue
            return found
    

    findall的逻辑是:

    ① 检查kwargs字典key有无'name',有则加入查询条件search_opts。
    ② 检查kwargs字典key有无'display_name',有则加入查询条件search_opts。
    ③ self.list(search_opts=search_opts) 用查询条件查询VolumeManager.list。如果前两步都为否,则查询条件为空,那么list方法将查询所有卷信息返回。cinder的db api有配置单次查询最大条数是1000(配置项osapi_max_limit,默认1000),而我们的卷目前有8k多条,其中2528条可用的,所以得查三次才能查出所有卷记录。这占用时间很长。
    ④ 对第三步查出的卷列表做遍历,用kwargs字典的key和value检查,符合getattr(obj, attr) == value的记录添加进found数组返回。

    5.human_id说明

    上文介绍cinderclient.utils.find_volume方法里用{human_id:name_or_id}作为条件调用VolumeManager.find查询。human_id是什么?我们可以看到cinderclient.v3.volumes.Volume继承自cinderclient.apiclient.base.Resource,而Resource类里定义了HUMAN_ID = FalseNAME_ATTR = 'name',Resource类里还有一个human_id(self)属性方法。

    cinderclient.apiclient.base.Resource

    class Resource(RequestIdMixin):
        """Base class for OpenStack resources (tenant, user, etc.).
        This is pretty much just a bag for attributes.
        """
        HUMAN_ID = False
        NAME_ATTR = 'name'
        
        @property
        def human_id(self):
            """Human-readable ID which can be used for bash completion.
            """
            if self.NAME_ATTR in self.__dict__ and self.HUMAN_ID:
                return strutils.to_slug(getattr(self, self.NAME_ATTR))
            return None
    

    根据human_id(self)属性方法的逻辑,我们可以这么理解:如果在cinderclient.v3.volumes.Volume类里有定义HUMAN_ID = True,且定义了类的名字属性NAME_ATTR ,那么volume.human_id=strutils.to_slug(getattr(self, self.NAME_ATTR))strutils.to_slug这个方法的作用是把参数转换成小写、移除non-word的字符(比如中文)、空格转连字符(-)、删除前后端空格。

    那么现在我们基本明白human_id的作用了:

    human_id是用于查询字符格式不标准的NAME_ATTR 的记录。db里可能存的NAME_ATTR包含了非不正常的字符,比如中文啊、多出的空格啊,cinderclient指令无法传递正确编码的参数到服务端查询。所以通过volume.human_id=strutils.to_slug(getattr(self, self.NAME_ATTR))方法设置human_id的值为NAME_ATTR(即name)转换格式后的值,这样对human_id来查询,就能查到了。比如下面的例子:
    数据库里有个volume的display_name= miaomiao喵喵猫,


    image.png

    但是在linux显示乱码:


    image.png

    如果直接使用” cinder show miaomiao喵喵猫 ” 来查询,会提示错误 “ERROR: 'utf8' codec can't decode byte 0xdf in position 8: invalid continuation byte”,无法使用name来查询:


    image.png

    但只要我们在cinderclient.v3.volumes.Volume类里设置HUMAN_ID=True,strutils.to_slug(u'miaomiao喵喵猫') == 'miaomiao',即可使用`”cinder show miaomiao”查到结果。

    image.png

    但是我们看cinderclient.v3.volumes.Volume没有复写HUMAN_ID,根据属性方法 human_id(self)的逻辑,如果self.HUMAN_ID==False,返回None作为volume的human_id属性值。所以,如果用{human_id:name_or_id}作为条件调用VolumeManager.find查询,getattr(obj, attr) == value是一定为False,find结果一定为空。

    既然cinderclient.v3.volumes.Volume的human_id=None,为什么utils.find_resource(manager, name_or_id, **kwargs) 最后一步还要用human_id查询呢?用human_id查询是不是有什么特别的用途?
    我用HUMAN_ID搜索cinderclient整个项目,发现只有cinderclient.apiclient.base.Resource里有用到,其它Resource的子类都没有复写。

    image.png

    用human_id搜索cinderclient整个项目,也没用对human_id特别赋值的地方。


    image.png

    这样看来,HUMAN_ID在cinderclient整个项目里的调用值都为False,human_id将始终未None,用{human_id:name_or_id}作为条件调用VolumeManager.find查询在当前的cinderclient里实在毫无意义。

    解决方案:

    可以考虑把{human_id:name_or_id}查询屏蔽。

    相关文章

      网友评论

          本文标题: “Cinder show 不存在的id,查询时间很长”分析

          本文链接:https://www.haomeiwen.com/subject/aodnixtx.html