环境路由和floatingip均为分布式的

东西向二层流量

OVN_EW_L2.png
同一geneve网络下跨节点访问

vlan网络net1下的云主机vm1访问vm3（上图标号1.*）

vm1的流量到br-int走逻辑交换机，打上vlan标签从br-int和br-prv的patch口发出(每个网络OVN都会自动创建一个patch口)

 cookie=0xefb298f1, duration=359369.164s, table=65, n_packets=79, n_bytes=7086, idle_age=12740, hard_age=65534, priority=100,reg15=0x1,metadata=0x1f actions=mod_vlan_vid:1157,output:681,strip_vlan

外层交换机转发后数据从patch口进入后剥离vlan，并设置metadata等数据后转发到vm3

 cookie=0xefb298f1, duration=345143.445s, table=0, n_packets=74, n_bytes=7084, idle_age=65534, hard_age=65534, priority=150,in_port=1125,dl_vlan=1157 actions=strip_vlan,load:0x61->NXM_NX_REG13[],load:0x5f->NXM_NX_REG11[],load:0x60->NXM_NX_REG12[],load:0x1f->OXM_OF_METADATA[],load:0x1->NXM_NX_REG14[],resubmit(,8)

数据包带vlan tag，mac地址为vm1和vm3的mac地址

11:45:55.426855 fa:16:3e:d7:41:89 > fa:16:3e:72:29:33, ethertype 802.1Q (0x8100), length 102: vlan 1157, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 20933, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.112.13 > 192.168.112.76: ICMP echo request, id 63233, seq 0, length 64

overlay网络net2下的云主机vm2访问vm4（上图标号2.*）

vm2的流量到br-int走逻辑交换机，根据metadata设置vni id并封装geneve数据包后发出

 cookie=0xd33701a1, duration=349.008s, table=37, n_packets=10, n_bytes=980, idle_age=142, priority=100,reg15=0x3,metadata=0x20 actions=load:0x20->NXM_NX_TUN_ID[0..23],set_field:0x3->tun_metadata0,move:NXM_NX_REG14[0..14]->NXM_NX_TUN_METADATA0[16..30],output:548

datapath的tunnel_key

()[root@ovn-ovsdb-sb-0 /]# ovn-sbctl list datapath lc_net2
_uuid               : e3344b60-dabb-4426-b937-00f96bf5f7e6
external_ids        : {logical-switch="86faac9e-8a62-4ade-875a-3672add713c4", name=neutron-f3586ec0-8dbc-478d-9460-6f7784a42309, name2=lc_net2}
load_balancers      : []
tunnel_key          : 32

数据包带vni号，数值和metadata及datapath的tunnel_key一致，内层mac地址为vm2和vm4的mac。

11:42:35.735756 d6:35:ee:b0:05:48 > ea:d1:f1:27:dd:4c, ethertype IPv4 (0x0800), length 156: (tos 0x0, ttl 64, id 62704, offset 0, flags [DF], proto UDP (17), length 142)
    192.168.20.2.ardus-cntl > 192.168.20.4.6081: [bad udp cksum 0xa9e2 -> 0x076e!] Geneve, Flags [C], vni 0x20, proto TEB (0x6558), options [class Open Virtual Networking (OVN) (0x102) type 0x80(C) len 8 data 00030002]
        fa:16:3e:5b:fe:bb > fa:16:3e:7d:b1:b4, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 54155, offset 0, flags [DF], proto ICMP (1), length 84)
    10.66.3.139 > 10.66.0.177: ICMP echo request, id 58369, seq 7, length 64

东西向三层流量

OVN_EW_L3.png
不同子网虚机跨节点访问，net1为geneve网络，net2为vlan网络 不同子网虚机跨节点访问，两个网络均为geneve网络

overlay网络net2下的云主机vm2访问vlan网络下net1下的云主机vm3

vm2的流量到br-int走逻辑路由，后打上vlan标签从br-int的patch口发出。如果目的主机所在主机使用的geneve，则逻辑路由后封装成geneve包。

 cookie=0xefb298f1, duration=338936.038s, table=65, n_packets=51, n_bytes=4998, idle_age=4107, hard_age=65534, priority=150,reg15=0x1,metadata=0x1f,dl_src=fa:16:3e:6c:a0:2e actions=mod_dl_src:ce:05:16:96:4d:43,mod_vlan_vid:1157,output:681

数据包

14:09:50.794428 ce:05:16:96:4d:43 > fa:16:3e:72:29:33, ethertype 802.1Q (0x8100), length 102: vlan 1157, p 0, ethertype IPv4, (tos 0x0, ttl 63, id 37976, offset 0, flags [DF], proto ICMP (1), length 84)
    10.66.3.139 > 192.168.112.76: ICMP echo request, id 60673, seq 3, length 64

南北向流量SNAT模式

OVN_SN_SNAT.png
geneve网络下虚机访问外部网络（lrp port和虚机在不同一节点） geneve网络下虚机访问外部网络（lrp port和虚机在同一节点）

vlan网络net1下的云主机vm1访问外部ip，但是逻辑路由的lrp port不在vm1所在的节点

逻辑路由信息

()[root@ovn-ovsdb-nb-2 /]# ovn-nbctl show 9eb2766d-dec8-4f73-8bc8-9f31dab3d6db
router 9eb2766d-dec8-4f73-8bc8-9f31dab3d6db (neutron-62e2ca59-b6cf-4a17-a531-776fecd8dbf5) (aka lc_router)
    port lrp-f60cd81a-7a5e-4b2d-b687-fcf2881a725a
        mac: "fa:16:3e:fd:11:6c"
        networks: ["10.66.0.1/16"]
    port lrp-cc3da0db-31e8-40a0-ad9f-c54f7a848ac7
        mac: "fa:16:3e:e7:f2:d2"
        networks: ["172.90.0.111/24"]
        gateway chassis: [9e7fc81f-12dc-4e83-b28c-a90060c579c6 ef5bb610-b0e6-4dc4-aa59-6d4d7972bc22 e8427df8-1dc6-45e3-b45c-35c35e8d6ed3]
    port lrp-d14d66bc-dfda-44a2-b1aa-a0481c0119ab
        mac: "fa:16:3e:6c:a0:2e"
        networks: ["192.168.112.1/24"]
    nat 1b9b075b-e56d-45e7-a750-cb1e366a7d08
        external ip: "172.90.0.111"
        logical ip: "192.168.112.0/24"
        type: "snat"
    nat 1d0eb0c0-67e8-43a3-8f7d-4da54d3fa562
        external ip: "172.90.0.111"
        logical ip: "10.66.0.0/16"
        type: "snat"
    nat bc8472b4-3ef9-4a18-bc6b-e9f12db01d4d
        external ip: "172.90.0.118"
        logical ip: "10.66.3.139"
        type: "dnat_and_snat"

lrp-cc3da0db-31e8-40a0-ad9f-c54f7a848ac7落在的节点信息(lrp port在node-3上，但是vm1在node-1)

()[root@ovn-ovsdb-sb-0 /]# ovn-sbctl show | grep -B 9 lrp-cc3da0db-31e8-40a0-ad9f-c54f7a848ac7
Chassis "9e7fc81f-12dc-4e83-b28c-a90060c579c6"
    hostname: node-3.domain.tld
    Encap geneve
        ip: "192.168.20.4"
        options: {csum="true"}
    Port_Binding "a7578b9a-c0b6-4980-8961-c968e027926f"
    Port_Binding "a9dce74f-f655-42b8-9529-f26a370feca9"
    Port_Binding "3427338f-1e5a-494b-9a8c-34bc80590c61"
    Port_Binding "f57f5ac7-6491-416f-8eb5-e4d1cb5fd41f"
    Port_Binding cr-lrp-cc3da0db-31e8-40a0-ad9f-c54f7a848ac7

数据包路径(vlan网络为例):

vm1->br-int(node-1)->br-ex(node-1)->br-ex(node-3)->br-int(node-3)->br-ex(node-3)->public network->target ip->br-ex(node-3)->patch port to br-prv->private network->br-prv->patch port to br-int->br-int->vm1

1. vm1出来的数据包，先经过本节点的逻辑路由发往lrp port（sip: ip vm1, dip: dst ip, smac: mac chassis which vm1 on, dmac: mac lrp port）并带上public network的vlan号

16:48:17.822115 6a:22:16:11:a3:47 > fa:16:3e:e7:f2:d2, ethertype 802.1Q (0x8100), length 102: vlan 2901, p 0, ethertype IPv4, (tos 0x0, ttl 63, id 39850, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.112.13 > 14.215.177.39: ICMP echo request, id 56321, seq 25, length 64

2. lrp port所在节点的逻辑路由将sip和smac转换成lrp port的ip和mac后通过br-ex发出（sip: ip lrp port, dip: dst ip, smac: mac lrp port）

16:48:17.822127 fa:16:3e:e7:f2:d2 > f8:bc:12:4e:44:dd, ethertype 802.1Q (0x8100), length 102: vlan 2901, p 0, ethertype IPv4, (tos 0x0, ttl 62, id 39850, offset 0, flags [DF], proto ICMP (1), length 84)
    172.90.0.111 > 14.215.177.39: ICMP echo request, id 56321, seq 25, length 64

3. 数据回包时lrp port所在的逻辑路由收到数据包（sip: target ip, dip: ip lrp port, dmac: mac lrp port）后直接路由，将数据包打上vlan后通过br-prv发出（sip: target ip, dip: ip vm1, dmac: mac vm1）

16:48:17.858433 00:1d:09:65:eb:63 > fa:16:3e:e7:f2:d2, ethertype 802.1Q (0x8100), length 102: vlan 2901, p 0, ethertype IPv4, (tos 0x0, ttl 51, id 39850, offset 0, flags [DF], proto ICMP (1), length 84)
    14.215.177.39 > 172.90.0.111: ICMP echo reply, id 56321, seq 25, length 64
16:48:17.858890 f2:d0:90:45:36:48 > fa:16:3e:d7:41:89, ethertype 802.1Q (0x8100), length 102: vlan 1157, p 0, ethertype IPv4, (tos 0x0, ttl 50, id 39850, offset 0, flags [DF], proto ICMP (1), length 84)
    14.215.177.39 > 192.168.112.13: ICMP echo reply, id 56321, seq 25, length 64

南北向流量floagintip（SNAT and DNAT）模式

OVN_SN_SNAT_DNAT.png
通过floatingip访问外部网络

vm1上挂有floatingip访问外部网络ip，不管逻辑路由的lrp port在哪个节点

vm1出去的和到vm1的数据包到br-int的逻辑路由后直接做SNAT和DNAT

14:54:25.282304 fa:16:3e:b5:dc:a7 > fa:16:3e:e7:f2:d2, ethertype 802.1Q (0x8100), length 102: vlan 2901, p 0, ethertype IPv4, (tos 0x0, ttl 63, id 8994, offset 0, flags [DF], proto ICMP (1), length 84)
    172.90.0.118 > 14.215.177.39: ICMP echo request, id 61441, seq 0, length 64
14:54:25.317236 00:1d:09:65:eb:63 > fa:16:3e:b5:dc:a7, ethertype 802.1Q (0x8100), length 102: vlan 2901, p 0, ethertype IPv4, (tos 0x0, ttl 51, id 8994, offset 0, flags [DF], proto ICMP (1), length 84)
    14.215.177.39 > 172.90.0.118: ICMP echo reply, id 61441, seq 0, length 64