arp_cache: neighbor table overfl

由于之前采用了静态arp，而静态arp在设计上很难维护清理的问题，因为所有节点都有静态arp，而只有一个节点删除pod，所以几乎无法实现静态arp清理的逻辑。 所以肯定会导致有一部分arp持久占用arp对应的内存资源。

[root@europe-worker-4 ~]# arp -a -n | grep PERM | wc -l
660
[root@europe-worker-4 ~]# arp -a -n | wc -l
663

可以看到静态arp几乎占了99.999 绝大部分的数量，其他都是临时arp记录。关于arp的限制由如下系统参数控制

/proc/sys/net/ipv4/neigh/default/gc_thresh1 - INTEGER
    Minimum number of entries to keep.  Garbage collector will not
    purge entries if there are fewer than this number.
    Default: 128

# 可以看到arp记录明显大于128 所以会触发清理，这个清理频率可能会比较低

/proc/sys/net/ipv4/neigh/default/gc_thresh2 - INTEGER
    Threshold when garbage collector becomes more aggressive about
    purging entries. Entries older than 5 seconds will be cleared
    when over this number.
    Default: 512

# 可以看到arp记录明显大于512 所以会触发更激进的清理机制，比如5s清理一次


/proc/sys/net/ipv4/neigh/default/gc_thresh3 - INTEGER
    Maximum number of non-PERMANENT neighbor entries allowed.  Increase
    this when using large numbers of interfaces and when communicating
    with large numbers of directly-connected peers.
    Default: 1024 

# 允许永久arp的最大上限

解决方式

 cat /etc/sysctl.conf
# sysctl settings are defined through files in
# /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
#
# Vendors settings live in /usr/lib/sysctl.d/.
# To override a whole file, create a new file with the same in
# /etc/sysctl.d/ and put new settings there. To override
# only specific settings, add a file with a lexically later
# name in /etc/sysctl.d/ and put new settings there.
#
# For more information, see sysctl.conf(5) and sysctl.d(5).
#
 ## works best with <= 500 client computers ##
# Force gc to clean-up quickly
net.ipv4.neigh.default.gc_interval = 3600

# Set ARP cache entry timeout
net.ipv4.neigh.default.gc_stale_time = 3600

# Setup DNS threshold for arp
net.ipv4.neigh.default.gc_thresh3 = 4096
net.ipv4.neigh.default.gc_thresh2 = 2048
net.ipv4.neigh.default.gc_thresh1 = 1024


# 执行 sysctl -p

如果cni daemon 可以恢复没写入的永久arp的话，可以重启下服务恢复。

总结: 静态arp的维护上的槽点还是比较明显的，可以考虑基于免费arp来优化。

参考：
https://phuoctaihuynh.wordpress.com/2020/04/09/neighbour-arp_cache-neighbor-table-overflow/
https://www.server24.eu/private-cloud/solve-neighbor-table-overflow-messages/
https://www.server24.eu/private-cloud/solve-neighbor-table-overflow-messages/