现象
虚拟机运行正常，但是 power_state 为 NOSTATE
[root@10e129e169e140 ~] openstack server show 050710bb-dbc5-44f2-aec8-7842fd0eac63
+-------------------------------------+-----------------------------------------------------------+
| Field | Value |
+-------------------------------------+-----------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | hp_001 |
| OS-EXT-SRV-ATTR:host | 10e129e170e11 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | 10e129e170e11 |
| OS-EXT-SRV-ATTR:instance_name | instance-050710bb-dbc5-44f2-aec8-7842fd0eac63 |
| OS-EXT-STS:power_state | NOSTATE |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2018-06-29T10:07:31.000000
| addresses | bj-net=192.168.9.17; ext-net-cn2=10.251.49.114 |
| config_drive | |
| created | 2018-06-29T10:07:21Z |
| flavor | 40C128G1600GEPH (138ab489-cc91-43ed-b392-44b2277f1371) |
| hostId | b47e6782505b685d311d1a2d0dabd6b1bf4c51437bafd1bc41d8c86f |
| id | 050710bb-dbc5-44f2-aec8-7842fd0eac63 |
| image | centos72-zabbix (21c988d5-e700-486f-bc22-782a1cab2a7a) |
| key_name | bj |
| name | 192e168e9e17 |
| progress | 98 |
| project_id | 936444f7909e4757bd88eea04e9a9550 |
| properties | nic_multiqueue_enabled='true' |
| security_groups | id='b3418b96-6bc4-4117-9f9d-19e40e913db8', name='default' |
| status | ACTIVE |
| updated | 2018-08-06T08:00:46Z |
| user_id | b89c47694bad4fb28383a32325a7de3b |
| volumes_attached | id='e86300f5-e268-4464-b68d-9d1c0e3bec51' |
compute 日志发现 InstanceNotFound
2018-08-06 16:02:05.346 13910 WARNING nova.compute.manager [req-a5db4fe4-250d-4795-9bbc-08be341dc900 8a974b0ae27f40ee87a92916f909360d c0f1fa6073be435491e4309ce0be959b - default default] [instance: 050710bb-dbc5-44f2-aec8-7842fd0eac63] Instance is unexpectedly not found. Ignore.: InstanceNotFound: Instance 050710bb-dbc5-44f2-aec8-7842fd0eac63 could not be found.
2018-08-06 16:02:05.347 13910 DEBUG oslo_concurrency.lockutils [req-a5db4fe4-250d-4795-9bbc-08be341dc900 8a974b0ae27f40ee87a92916f909360d c0f1fa6073be435491e4309ce0be959b - default default] Lock "050710bb-dbc5-44f2-aec8-7842fd0eac63" released by "nova.compute.manager.query_driver_power_state_and_sync" :: held 0.081s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285
manager.py 捕获到 InstanceNotFound 设置 vm_power_state=NOSTATE
file: /usr/lib/python2.7/site-packages/nova/compute/manager.py
def _query_driver_power_state_and_sync(self, context, db_instance):
if db_instance.task_state is not None:
LOG.info("During sync_power_state the instance has a "
"pending task (%(task)s). Skip.",
{'task': db_instance.task_state}, instance=db_instance)
return
# No pending tasks. Now try to figure out the real vm_power_state.
try:
vm_instance = self.driver.get_info(db_instance)
vm_power_state = vm_instance.state
except exception.InstanceNotFound: "捕获InstanceNotFound"
vm_power_state = power_state.NOSTATE
# Note(maoy): the above get_info call might take a long time,
# for example, because of a broken libvirt driver.
try:
self._sync_instance_power_state(context,
db_instance,
vm_power_state,
use_slave=True)
except exception.InstanceNotFound:
# NOTE(hanlind): If the instance gets deleted during sync,
# silently ignore.
pass
nova@localhost 17:00: [(none)]> select * from nova.instances where \n uuid='050710bb-dbc5-44f2-aec8-7842fd0eac63'\G
*************************** 1. row ***************************
created_at: 2018-06-29 10:07:21
updated_at: 2018-08-06 08:00:46
deleted_at: NULL
id: 1822
user_id: b89c47694bad4fb28383a32325a7de3b
project_id: 936444f7909e4757bd88eea04e9a9550
launch_index: 0
key_name: bj
power_state: 0
vm_state: active
memory_mb: 131072
vcpus: 40
hostname: 192e168e9e17
host: 10e129e170e11
launched_at: 2018-06-29 10:07:31
terminated_at: NULL
display_name: 192e168e9e17
display_description: NULL
availability_zone: hp_001
locked: 0
os_type: linux
launched_on: 10e129e170e26
uuid: 050710bb-dbc5-44f2-aec8-7842fd0eac63
node: 10e129e170e11
deleted: 0
cleaned: 1
1 row in set (0.00 sec)
nova@localhost 17:00: [(none)]>
数据库记录在 10e129e170e11 节点，实际 Instance 启动在10e129e170e41 节点
[root@10e129e169e140 ~]# nova instance-action-list 050710bb-dbc5-44f2-aec8-7842fd0eac63
+------------------+------------------------------------------+---------+----------------------------+----------------------------+
| Action | Request_ID | Message | Start_Time | Updated_At |
+------------------+------------------------------------------+---------+----------------------------+----------------------------+
| create | req-474eec18-92ae-4b66-86b0-18a2bbd08806 | - | 2018-06-29T10:07:21.000000 | 2018-06-29T10:07:32.000000 |
| attach_volume | req-634dc62c-d26f-4867-9ab3-13b947edf266 | - | 2018-06-29T10:12:48.000000 | 2018-06-29T10:12:50.000000 |
| attach_interface | req-45778f28-28f4-4e86-a0e7-88c326372a09 | - | 2018-06-29T10:46:32.000000 | 2018-06-29T10:46:36.000000 |
| live-migration | req-cc2eb8dd-8e94-477a-a91d-aab0699a815d | Error | 2018-07-12T10:01:47.000000 | 2018-07-12T10:01:50.000000 |
+------------------+------------------------------------------+---------+----------------------------+----------------------------+
[root@10e129e170e41 ~] virsh list --all
Id Name State
----------------------------------------------------
12 instance-050710bb-dbc5-44f2-aec8-7842fd0eac63 running
13 instance-fcb96c44-340e-4663-83a1-5e999246833f running
原因
由于之前 migrate ERROR 导致 instance 实际启动的节点和数据库记录的节点不一致，comoute 通过 manager.py 捕获到 InstanceNotFound， 所以把 vm_power_state 状态置为 NOSTATE
解决方法
重新同步数据库，硬重启虚拟机
网友评论