在 OpenStack 中，虚拟机获取 Metadata 信息的方式有两种：Config drive 和 metadata RESTful 服务。下面对通过 metadata RESTful 服务获取元数据分析
通过 router 发送请求 （当 subnet 连接了 router 时 ）
虚拟机内部 curl metadata
[root@vm-router~] curl http://169.254.169.254/2009-04-04/meta-data
ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
hostname
instance-action
instance-id
instance-type
local-hostname
local-ipv4
placement/
public-hostname
public-ipv4
reservation-id
[root@vm-router ~] route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.10.10.1 0.0.0.0 UG 0 0 0 eth0
10.10.10.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
"169.254.169.254 10.10.10.1 255.255.255.255 UGH 0 0 0 eth0"
虚拟机内部会把 169.254.169.254 发向 10.10.10.1 ( network 路由网关)
[root@network ~] ip netns exec qrouter-d02c9901-d249-4189-a286-1477b7522d0a iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N neutron-l3-agent-OUTPUT
-N neutron-l3-agent-POSTROUTING
-N neutron-l3-agent-PREROUTING
-N neutron-l3-agent-float-snat
-N neutron-l3-agent-snat
-N neutron-postrouting-bottom
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
"-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697"
-A neutron-postrouting-bottom -m comment --comment Perform source NAT on outgoing traffic -j neutron-l3-agent-snat
[root@network ~] grep -r add-header /var/lib/neutron/ns-metadata-proxy/d02c9901-d249-4189-a286-1477b7522d0a.conf
http-request add-header X-Neutron-Router-ID d02c9901-d249-4189-a286-1477b7522d0a
ns-metadata-proxy, add-header X-Neutron-Router-ID 并通过在 router 所在网络命名空间添加 iptables 规则，将该报文转发至 9697 端口 ( neutron-ns-metadata-proxy )
[root@network ~] ip netns exec qrouter-d02c9901-d249-4189-a286-1477b7522d0a netstat -anlp |grep 9697
tcp 0 0 0.0.0.0:9697 0.0.0.0:* LISTEN 114178/haproxy
[root@network ~] ip netns exec qrouter-d02c9901-d249-4189-a286-1477b7522d0a ps -ef |grep 114178
neutron 114178 1 0 Jun22 ? 00:00:06 haproxy -f /var/lib/neutron/ns-metadata-proxy/d02c9901-d249-4189-a286-1477b7522d0a.conf
root 438102 323147 0 13:56 pts/28 00:00:00 grep --color=auto 114178
neutron-ns-metadata-proxy 通过 Unix domain socket 将请求发给 neutron-metadata-agent
metadata-agent 的请求日志信息
[root@vm-router ~] curl http://169.254.169.254/2009-04-04/meta-data
[root@network ~] tailf /var/log/neutron/metadata-agent.log
2018-06-25 16:35:29.120 68237 DEBUG eventlet.wsgi.server [-] (68237) accepted '' server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:883
2018-06-25 16:35:29.122 68237 DEBUG neutron.agent.metadata.agent [-] Request: GET /2009-04-04/meta-data HTTP/1.0^M
Accept: */*^M
Connection: close^M
Content-Type: text/plain^M
Host: 169.254.169.254^M
User-Agent: curl/7.29.0^M
X-Forwarded-For: 10.10.10.12^M
X-Neutron-Router-Id: d02c9901-d249-4189-a286-1477b7522d0a __call__ /usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py:86
2018-06-25 16:35:29.211 68237 DEBUG neutron.agent.metadata.agent [-] Gotten ports for remote_address 10.10.10.12, network_id None, router_id d02c9901-d249-4189-a286-1477b7522d0a are: [{u'allowed_address_pairs': [], u'extra_dhcp_opts': [], u'updated_at': u'2018-06-22T02:30:30Z', u'device_owner': u'compute:nova', u'revision_number': 10, u'port_security_enabled': True, u'binding:profile': {}, u'fixed_ips': [{u'subnet_id': u'bf017706-438d-4c56-b41a-8bcf5243dc1d', u'ip_address': u'10.10.10.12'}], u'id': u'cbfca678-e4ca-401f-bd58-89fdc8f04863', u'security_groups': [u'ba8284a1-280f-4fc4-a576-8d880c196618'], u'binding:vif_details': {u'port_filter': True, u'datapath_type': u'system', u'ovs_hybrid_plug': False}, u'binding:vif_type': u'ovs', u'qos_policy_id': None, u'mac_address': u'fa:16:3e:ab:8b:b6', u'project_id': u'3ae80176a60f4d989c84c084ca05df1c', u'status': u'ACTIVE', u'binding:host_id': u'10e129e168e38', u'description': u'', u'tags': [], u'dns_assignment': [{u'hostname': u'host-10-10-10-12', u'ip_address': u'10.10.10.12', u'fqdn': u'host-10-10-10-12.openstacklocal.'}], u'device_id': u'a3951248-2418-4ee2-9f2c-efca143b2a21', u'name': u'', u'admin_state_up': True, u'network_id': u'db6039d8-2a48-4119-8f74-2441dae76e3a', u'dns_name': u'', u'created_at': u'2018-06-22T02:30:06Z', u'binding:vnic_type': u'normal', u'tenant_id': u'3ae80176a60f4d989c84c084ca05df1c'}] _get_instance_and_tenant_id /usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py:166
2018-06-25 16:35:29.369 68237 DEBUG neutron.agent.metadata.agent [-] {'date': 'Mon, 25 Jun 2018 08:35:29 GMT', 'status': '200', 'content-length': '210', 'content-type': 'text/plain; charset=UTF-8', 'content-location': u'http://nova-api-metadata:10012/2009-04-04/meta-data'} _proxy_request /usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py:203
2018-06-25 16:35:29.370 68237 INFO eventlet.wsgi.server [-] 10.10.10.12,<local> "GET /2009-04-04/meta-data HTTP/1.1" status: 200 len: 347 time: 0.2482362
通过 DHCP 发送请求 （当 subnet 未连接 router 时）
[root@no_router ~] route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.10.20.1 0.0.0.0 UG 0 0 0 eth0
10.10.20.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
169.254.169.254 10.10.20.2 255.255.255.255 UGH 0 0 0 eth0
[root@no_router ~] curl http://169.254.169.254/2009-04-04/meta-data
ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
hostname
instance-action
instance-id
instance-type
local-hostname
local-ipv4
placement/
public-hostname
public-ipv4
reservation-id
security-groups
虚拟机内部会把 169.254.169.254 发向 qdhcp
[root@network ~] grep -r add-header /var/lib/neutron/ns-metadata-proxy/20a738d6-f885-4383-b73f-616b588849c6.conf
http-request add-header X-Neutron-Network-ID 20a738d6-f885-4383-b73f-616b588849c6
[root@network ~] ip netns exec qdhcp-20a738d6-f885-4383-b73f-616b588849c6 netstat -anlp |grep 80
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 5226/haproxy
tcp6 0 0 fe80::f816:3eff:fece:53 :::* LISTEN 288823/dnsmasq
udp6 0 0 fe80::f816:3eff:fece:53 :::* 288823/dnsmasq
[root@network ~] ip netns exec qdhcp-20a738d6-f885-4383-b73f-616b588849c6 ps -ef |grep 5226
neutron 5226 1 0 Jun21 ? 00:00:08 haproxy -f /var/lib/neutron/ns-metadata-proxy/20a738d6-f885-4383-b73f-616b588849c6.conf
root 154079 323147 0 17:11 pts/28 00:00:00 grep --color=auto 5226
[root@10e129e168e38 ~]
[root@network ~] ip netns exec qdhcp-20a738d6-f885-4383-b73f-616b588849c6 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
222: tap93e7a16a-3d: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
link/ether fa:16:3e:ce:1e:ca brd ff:ff:ff:ff:ff:ff
inet 10.10.10.2/24 brd 10.10.10.255 scope global tap93e7a16a-3d
valid_lft forever preferred_lft forever
"inet 169.254.169.254/16 brd 169.254.255.255 scope global tap93e7a16a-3d"
valid_lft forever preferred_lft forever
inet 10.10.11.3/24 brd 10.10.11.255 scope global tap93e7a16a-3d
valid_lft forever preferred_lft forever
inet 10.10.12.3/24 brd 10.10.12.255 scope global tap93e7a16a-3d
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fece:1eca/64 scope link
valid_lft forever preferred_lft forever
后面情况和 使用router就一样了，neutron-ns-metadata-proxy 将请求通过 unix domain socket 发给 neutron-metadata-agent
两者情况对比
-
连接router情况，是 router netns 用 iptables 规则来转发，add-header X-Neutron-Router-ID 和 instance_ip,请求 169.254.169.254 的路由 发现 10.10.10.1（ router ）
-
未连接 router 情况，是 dhcp netns 将此 169.254.169.254 配置到自己的接口上,add-header X-Neutron-Network-ID 和 instance_ip,请求 169.254.169.254 的路由 发现 10.10.20.2（ dchp ）
大概过程总结
-
instance 通过 router 或者 dhcp 将 metadata 请求发送到 neutron-ns-metadata-proxy。
-
neutron-ns-metadata-proxy 会将 instance_ip 和 router_id 或者 network_id 添加到 http 请求的 head 通过 unix domain socket 将请求发给 neutron-metadata-agent。
-
neutron-metadata-agent 将根据 IP 和 router 得到 instance_id 添加到 http 请求的 head 中，然后转发给 nova-api-metadata 。
-
nova-api-metadata 从请求的 HTTP 头部中取出相应的信息，获得虚拟机的 ID，继而从数据库中读取虚拟机的 metadata 信息，最后将结果返回
网友评论