排查 nce/pubt1 has no valid networ

上层同学报过来的错误。

2017-02-22 14:31:22,960 76841196 [pool-6-thread-41] (NVSServiceImpl.java:539) WARN  - 93e30e33-d136-43bb-b5a2-9c1bd9f5519f is error during scheduling!
2017-02-22 14:31:22,960 76841196 [pool-6-thread-41] (NVSServiceImpl.java:549) INFO  - [createServerWaitAZ]create vm fail, begin to deleteing sid。93e30e33-d136-43bb-b5a2-9c1bd9f5519f
2017-02-22 14:31:22,966 76841202 [pool-6-thread-41] (HttpClientUtil.java:145) DEBUG - ==== uuid: 4f0227e625394c4e8f18464c54b0e485 , ReqURL: http://10.185.3.253:8774/v2/56ef8405224b44fea50ec80cbfce8fed/servers/93e30e33-d136-43bb-b5a2-9c1bd9f5519f
2017-02-22 14:31:22,966 76841202 [pool-6-thread-41] (HttpClientUtil.java:146) DEBUG - ==== uuid: 4f0227e625394c4e8f18464c54b0e485 , ReqMethod: DELETE
2017-02-22 14:31:22,966 76841202 [pool-6-thread-41] (HttpClientUtil.java:149) DEBUG - ==== uuid: 4f0227e625394c4e8f18464c54b0e485 , ReqHeader: X-Auth-Token = 29c9018665ef4a5db4066285d6ceacec
2017-02-22 14:31:22,966 76841202 [pool-6-thread-41] (HttpClientUtil.java:152) DEBUG - ==== uuid: 4f0227e625394c4e8f18464c54b0e485 , ReqBody: null
2017-02-22 14:31:23,121 76841357 [pool-6-thread-41] (HttpClientUtil.java:167) DEBUG - ==== uuid: 4f0227e625394c4e8f18464c54b0e485 , RespCode: 204
2017-02-22 14:31:23,121 76841357 [pool-6-thread-41] (HttpClientUtil.java:168) DEBUG - ==== uuid: 4f0227e625394c4e8f18464c54b0e485 , RespBody(startTime:1487745082966,endTime:1487745083121,elapse:155ms) Body: null

可见是云主机调度错误 uuid是这台 93e30e33-d136-43bb-b5a2-9c1bd9f5519f 云主机创建失败被回滚删了。

先去nova scheduler上查日志。

scheduler节点较多，要一个一个节点上去看日志。没有elk好苦

2017-02-22 14:31:21.894 173557 INFO nova.scheduler.filter_scheduler [req-b0ce6a30-03c0-4ceb-88d8-2d031b5f4f6c 8cd57421117640a0aa7012878dc53aa1 56ef8405224b44fea50ec80cbfce8fed] Choosing host WeighedHost [host: pubt1-nova21.yq.163.org, weight: 48709.0] for instance 93e30e33-d136-43bb-b5a2-9c1bd9f5519f

看到这，说明调度选择到 pubt1-nova21.yq.163.org 这个节点上去了。我们接下来跳到这个节点看nova compute 日志。

2017-02-22 14:31:22.804 22239 DEBUG nova.openstack.common.lockutils [req-b0ce6a30-03c0-4ceb-88d8-2d031b5f4f6c 8cd57421117640a0aa7012878dc53aa1 56
ef8405224b44fea50ec80cbfce8fed] Got semaphore "compute_resources" lock /usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py:167
2017-02-22 14:31:22.804 22239 DEBUG nova.openstack.common.lockutils [req-b0ce6a30-03c0-4ceb-88d8-2d031b5f4f6c 8cd57421117640a0aa7012878dc53aa1 56
ef8405224b44fea50ec80cbfce8fed] Got semaphore / lock "update_usage" inner /usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py:247
2017-02-22 14:31:22.822 22239 DEBUG nova.openstack.common.lockutils [req-b0ce6a30-03c0-4ceb-88d8-2d031b5f4f6c 8cd57421117640a0aa7012878dc53aa1 56
ef8405224b44fea50ec80cbfce8fed] Semaphore / lock released "update_usage" inner /usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.p
y:251
2017-02-22 14:31:22.841 22239 DEBUG neutronclient.client [-] RESP:{'date': 'Wed, 22 Feb 2017 06:31:22 GMT', 'status': '404', 'content-length': '1
03', 'content-type': 'application/json; charset=UTF-8'} {"NeutronError": {"message": "nce/pubt1 has no valid network", "type": "NoValidNetwork", 
"detail": ""}}
 http_log_resp /usr/lib/python2.7/dist-packages/neutronclient/common/utils.py:218
2017-02-22 14:31:22.841 22239 DEBUG neutronclient.v2_0.client [-] Error message: {"NeutronError": {"message": "nce/pubt1 has no valid network", "
type": "NoValidNetwork", "detail": ""}} _handle_fault_response /usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py:1388
2017-02-22 14:31:22.841 22239 ERROR nova.netease.network.multi_idc [req-b0ce6a30-03c0-4ceb-88d8-2d031b5f4f6c 8cd57421117640a0aa7012878dc53aa1 56e
f8405224b44fea50ec80cbfce8fed] Neutron error: Novalid network on vnetwork nce, az pubt1
2017-02-22 14:31:22.841 22239 TRACE nova.netease.network.multi_idc Traceback (most recent call last):
2017-02-22 14:31:22.841 22239 TRACE nova.netease.network.multi_idc   File "/usr/lib/python2.7/dist-packages/nova/netease/network/multi_idc.py", l
ine 149, in create_port_by_vnetwork
2017-02-22 14:31:22.841 22239 TRACE nova.netease.network.multi_idc     port_id = port_client.create_port(port_req_body)['port']['id']
2017-02-22 14:31:22.841 22239 TRACE nova.netease.network.multi_idc   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1
15, in with_params
2017-02-22 14:31:22.841 22239 TRACE nova.netease.network.multi_idc     ret = self.function(instance, *args, **kwargs)
2017-02-22 14:31:22.841 22239 TRACE nova.netease.network.multi_idc   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 3
44, in create_port
2017-02-22 14:31:22.841 22239 TRACE nova.netease.network.multi_idc     return self.post(self.ports_path, body=body)
2017-02-22 14:31:22.841 22239 TRACE nova.netease.network.multi_idc   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1
:

关键信息 nce/pubt1 has no valid network。nce网段没有。

使用neutron 查询一下

neutron vnet-binding-list --tenant-id 56ef8405224b44fea50ec80cbfce8fed
+--------------------------------------+----------+--------------------------------------+-------+
| id                                   | vnetwork | network_id                           | az    |
+--------------------------------------+----------+--------------------------------------+-------+
| 324d1d99-f8c2-4e02-abe4-79e0d158781a | public   | 2d80536d-46cb-4d25-9958-a4773fa3d7d6 | pubt1 |
| 86bcdadb-53ae-4f08-842c-f0d4e832406b | private  | 81331abc-397b-4680-b444-2b1f88749c80 | pubt1 |
| 92220458-f23e-47e8-bfcc-e9176e93ad7a | private  | 81331abc-397b-4680-b444-2b1f88749c80 | pub1  |
| a836b562-079b-405e-856f-fc31bf00c2e3 | public   | 74f67c3f-d1df-449e-938d-46b19080838d | pubt1 |
+--------------------------------------+----------+--------------------------------------+-------+

可见vnet中的确没有。

原因，初始化的时候没有走完nce的流程，让底层同学帮直接初始化了。

那解决的办法是可以帮他绑上一个nce的vnet。

先找个正常的通过nce初始化的租户，查询一下

neutron vnet-binding-list --tenant-id 6596d85821c54c16a70d919aa79c1bbc
+--------------------------------------+----------+--------------------------------------+-------+
| id                                   | vnetwork | network_id                           | az    |
+--------------------------------------+----------+--------------------------------------+-------+
| 07ea81fc-37a6-4ae0-8e95-8e2c48a854c6 | public   | 74f67c3f-d1df-449e-938d-46b19080838d | pubt1 |
| 4335aed2-4b0b-4b0a-8410-33440831a48e | nce      | 5646d40e-637c-4ce4-b306-cb5e62648a9c | pubt1 |
| b96f2bde-9345-4be3-a87f-96ab260cbffd | private  | d3ed1a4a-b4b9-430e-b147-180472fc3aeb | pubt1 |
+--------------------------------------+----------+--------------------------------------+-------+

然后帮他绑上

neutron vnet-binding-create --tenant-id 56ef8405224b44fea50ec80cbfce8fed --vnet nce    --network   5646d40e-637c-4ce4-b306-cb5e62648a9c    --az pubt1

done！