Linux bonding介绍

网卡bond是通过把多个物理网卡绑定为一个逻辑网卡，实现本地网卡的冗余，带宽扩容和负载均衡。
bonding技术的最早应用是在集群——beowulf上，为了提高集群节点间的数据传输而设计的。下面我们讨论一下bonding 的原理, 什么是bonding需要从网卡的混杂(promisc)模式说起。我们知道，在正常情况下，网卡只接收目的硬件地址(MAC Address)是自身Mac的以太网帧，对于别的数据帧都滤掉，以减轻驱动程序的负担。但是网卡也支持另外一种被称为混杂promisc的模式，可以接收网络 上所有的帧，比如说tcpdump，就是运行在这个模式下。bonding也运行在这个模式下，而且修改了驱动程序中的mac地址，将两块网卡的Mac地址改成相同，可以接收特定mac的数据帧。然后把相应的数据帧传送给bond驱动程序处理。
可以通过以下命令确定内核是否支持bonding：

(ENV) [root@ceph-2 ~]# cat /boot/config-`uname -r` | grep -i bonding
CONFIG_BONDING=m

bond配置后网卡信息如下，从输出可以看出系统中配置了两个bond，分别为bond0和bond1，bond0由enp103s0f0和enp103s0f1组成，bond1由ens1f0和ens1f1组成。bond中的网卡具有相同的mac地址。

(ENV) [root@ceph-2 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp103s0f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 5c:c9:99:6a:dc:64 brd ff:ff:ff:ff:ff:ff
3: enp103s0f1: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master bond0 state DOWN group default qlen 1000
    link/ether 5c:c9:99:6a:dc:64 brd ff:ff:ff:ff:ff:ff
4: ens1f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1 state UP group default qlen 1000
    link/ether 88:df:9e:35:bb:e8 brd ff:ff:ff:ff:ff:ff
5: ens1f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1 state UP group default qlen 1000
    link/ether 88:df:9e:35:bb:e8 brd ff:ff:ff:ff:ff:ff
6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 5c:c9:99:6a:dc:64 brd ff:ff:ff:ff:ff:ff
    inet 10.0.11.18/24 brd 10.0.11.255 scope global bond0
       valid_lft forever preferred_lft forever
    inet6 fe80::5ec9:99ff:fe6a:dc64/64 scope link
       valid_lft forever preferred_lft forever
7: bond1: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 88:df:9e:35:bb:e8 brd ff:ff:ff:ff:ff:ff
    inet 10.0.21.18/24 brd 10.0.21.255 scope global bond1
       valid_lft forever preferred_lft forever
    inet6 fe80::8adf:9eff:fe35:bbe8/64 scope link
       valid_lft forever preferred_lft forever

我们也可以通过查看/sys/class/net/目录下查看网卡信息。其目录下有个常规文件bonding_masters，记录bonding信息。其他的网卡文件都为链接文件。

sysfs is a virtual file system that represents kernel objects as directories, files and symbolic links. sysfs can be used to query for information about kernel objects, and can also manipulate those objects through the use of normal file system commands. The sysfs virtual file system is mounted under the /sys/ directory. All bonding interfaces can be configured dynamically by interacting with and manipulating files under the /sys/class/net/ directory.

(ENV) [root@ceph-2 net]# pwd
/sys/class/net
(ENV) [root@ceph-2 net]# ls
bond0  bond1  bonding_masters  enp103s0f0  enp103s0f1  ens1f0  ens1f1  lo
(ENV) [root@ceph-2 net]# cat bonding_masters
bond0 bond1

访问任一bond所对应的目录，可以看到其对应的sys文件系统信息。包括link_mode, operstate, type, speed, address等。如下所示为bond0的信息，也可以查看其下bonding目录下的信息，例如slaves, mode, miimon, mii_status等等。例子中mode=1 表示 fault-tolerance (active-backup)提供冗余功能，工作方式是主从的工作方式，也就是说默认情况下只有一块网卡工作，另一块做备份。

(ENV) [root@ceph-2 bond0]# pwd
/sys/class/net/bond0
(ENV) [root@ceph-2 bond0]# ls
addr_assign_type  carrier          duplex             iflink            netdev_group    power             speed         uevent
address           carrier_changes  flags              link_mode         operstate       proto_down        statistics
addr_len          dev_id           gro_flush_timeout  lower_enp103s0f0  phys_port_id    queues            subsystem
bonding           dev_port         ifalias            lower_enp103s0f1  phys_port_name  slave_enp103s0f0  tx_queue_len
broadcast         dormant          ifindex            mtu               phys_switch_id  slave_enp103s0f1  type
(ENV) [root@ceph-2 bond0]# cat operstate
up
(ENV) [root@ceph-2 bond0]# cat address
5c:c9:99:6a:dc:64
(ENV) [root@ceph-2 bond0]# cat speed
10000
(ENV) [root@ceph-2 bond0]# cat type
1
(ENV) [root@ceph-2 bond0]# cat link_mode
0
(ENV) [root@ceph-2 bond0]# cat iflink
6
(ENV) [root@ceph-2 bond0]# cat bonding/slaves
enp103s0f0 enp103s0f1
(ENV) [root@ceph-2 bond0]# cat bonding/mode
active-backup 1
(ENV) [root@ceph-2 bond0]# cat bonding/active_slave
enp103s0f0
(ENV) [root@ceph-2 bond0]# cat bonding/miimon
100
(ENV) [root@ceph-2 bond0]# cat bonding/mii_status
up

miimon=time_in_milliseconds

Specifies (in milliseconds) how often MII link monitoring occurs. This is useful if high availability is required because MII is used to verify that the NIC is active. To verify that the driver for a particular NIC supports the MII tool, type the following command as root:

~]# ethtool <interface_name> | grep "Link detected:"

In this command, replace interface_name> with the name of the device interface, such as eth0, not the bond interface. If MII is supported, the command returns:

Link detected: yes

If using a bonded interface for high availability, the module for each NIC must support MII. Setting the value to 0 (the default), turns this feature off. When configuring this setting, a good starting point for this parameter is 100.

(ENV) [root@ceph-2 bond0]# ethtool enp103s0f0
Settings for enp103s0f0:
    Supported ports: [ FIBRE ]
    Supported link modes:   10000baseT/Full
                            1000baseX/Full
                            10000baseSR/Full
                            10000baseLR/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  10000baseT/Full
                            1000baseX/Full
                            10000baseSR/Full
                            10000baseLR/Full
    Advertised pause frame use: No
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: 10000Mb/s
    Duplex: Full
    Port: FIBRE
    PHYAD: 0
    Transceiver: internal
    Auto-negotiation: off
    Supports Wake-on: g
    Wake-on: g
    Current message level: 0x00000007 (7)
                   drv probe link
    Link detected: yes

另外我们也可以通过proc文件系统来查看bond的信息

(ENV) [root@ceph-2 bond0]# ll /proc/net/bonding
total 0
-r--r--r-- 1 root root 0 Mar 14 23:56 bond0
-r--r--r-- 1 root root 0 Mar 14 23:56 bond1
(ENV) [root@ceph-2 bonding]# cat bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: enp103s0f0
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: enp103s0f0
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 5c:c9:99:6a:dc:64
Slave queue ID: 0

Slave Interface: enp103s0f1
MII Status: down
Speed: Unknown
Duplex: Unknown
Link Failure Count: 0
Permanent HW addr: 5c:c9:99:6a:dc:65
Slave queue ID: 0

网卡的统计信息可以通过/proc/net/dev查看

(ENV) [root@ceph-2 bonding]# cat /proc/net/dev
Inter-|   Receive                                                |  Transmit
 face |bytes    packets errs drop fifo frame compressed multicast|bytes    packets errs drop fifo colls carrier compressed
 bond1: 218380932618639 150903303281    0 254531    0     0          0       102 202196913483854 141663976093    0    0    0     0       0          0
 bond0: 4601316958197 4030868434    0  822    0     0          0   4681298 4903772722520 3881581069    0    0    0     0       0          0
enp103s0f0: 4601316958197 4030868434    0    0    0     0          0   4681298 4903772722520 3881581069    0    0    0     0       0          0
    lo: 13220872268767 2741169953    0    0    0     0          0         0 13220872268767 2741169953    0    0    0     0       0          0
ens1f0: 205136338566527 141871069749    0 37923    0     0          0        51 188931569389247 132611955898    0    0    0     0       0          0
enp103s0f1:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
ens1f1: 13244594130872 9032233628    0    0    0     0          0        51 13270685429231 9052021096    0    0    0     0       0          0
docker0:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0

References

• linux多网卡绑定bonding
• Linux网卡bond的七种模式详解
• networking_guide/sec-using_channel_bonding
• mii-tool
• Linux高性能计算集群 -- Beowulf集群
• 2.7. USING NETWORKMANAGER WITH SYSCONFIG FILES
• Introduction to Linux interfaces for virtual networking