While studying the traffic going to our core switches, we observed a high number of ARP packets getting into the Control Plane(CP) of the switches. The ARP broadcasts were very high and this was not an ideal behaviour under any circumstances. So, we wanted to know " What was the actual traffic hitting the CP of the switch?". We started capturing the packets going to the CP of the switch, to our surprise we observed that 50 to 60% of the total traffic going to the CP was the ARP Broadcast traffic. We then started to analyse why are we having such high ARP Broadcast traffic. We started going more granular on the issue. Switches had, by default, ARP timer of 4 hrs set on them. Before we go any further, lets see what an ARP Broadcast traffic means.
What is an ARP Broadcast Traffic? What is an ARP protocol?
The ARP (Address Resolution Protocol) is a broadcast protocol. ARP is used to convert a network address (e.g. an IPv4 address) to a physical address such as an Ethernet address (also known as a MAC address).
For example, when we try to ping an IP address 192.168.1.1, the system has to resolve the IP address into a MAC address. ARP resolves the IP address into a MAC address.
Every system keeps an ARP look-up table which consists of IP addresses and the associated MAC addresses. When we try to ping (or initiate communication) to an IP address on a locally attached network, the system would first consult its ARP table to see if it knows the MAC address of the IP. If it knows it, then ARP is not used.
If the IP address 192.168.1.1 is not found in the ARP table, the system will send out a broadcast packet to the network using the ARP protocol and ask for 192.168.1.1’s MAC address. Since its a broadcast packet this would be received by all hosts on the network and the host with the IP address 192.168.1.1 would reply with the MAC address of the IP 192.168.1.1.
Every host on the network monitors the network for these requests and when its address is requested it sends an arp reply. The arp reply specifies the 48 bit Ethernet address to use for that IP address.
The ARP resides at the Link layer of the TCP/IP model:
- TCP/IP Layers
- Application Layer
- Transport Layer
- Internet Layer
- Link Layer <----------------------------------------- ARP
Basic necessity of having an ARP?
Insufficient space to store MAC address along with the host part of the IP address in the IPv4 packet header.
How does this work in our environment?
Individual host machines maintain an ARP table which is used in conjunction with the MAC address table maintained by the switch for end to end communication. When the host machine’s ARP timer expires and if there is a need to talk to an IP address, the host will send out an ARP broadcast which will flood the network and this will also hit the control plane of the switch. When this request reaches the host which has the respective IP, it responds and the ARP table is updated based on the this reply.
There are various ways to see this traffic and one of the ways is to capture packets on the Core switches. When we did that, we were perplexed that 50% to 60% of the total traffic hitting the control plane of the switch was ARP traffic.
Now what ?
Checking further on why the servers were generating a high ARP broadcast traffic revealed that all of the machines had the ARP timer and ARP table size set to their default values (ARP timer of 30 seconds on Linux machines).
Why is it alarming? What happens when the ARP timer gets expired?
Too many ARP broadcasts can have an adverse effect on the network. This can eat up some of the network bandwidth, increase the CPU utilization on the switches and can cause various other issues in the network.
On all of the servers, the ARP table value was set to a very low value. This can be viewed in the sysctl config. Every time the ARP expires, the server sends an ARP Broadcast message requesting the MAC address(es) of all the IP address(es).
“Whenever the Server sees valid traffic coming from the corresponding device it resets the ARP age counter to zero for that particular IP address. This ensures that the addresses of active devices are never flushed out of the cache, no matter how long they have been known." So whenever there is no active communication and when the timer expires, and if server A wants to talk to server B, the server A sends out an ARP broadcast to get the MAC address of server B.
So what is the big deal?
Every time Server A wants to contact Server B and assuming no communication is established and the ARP timer has expired, the server will first have to send out an ARP broadcast frame, wait for the reply and only then can it start the communication. Isn’t it inducing latency in the network?
This problem wouldn’t be there if the ARP timer and the ARP table is relatively higher than the default values.
What we did?
We studied the configs on the servers and realised that there are a few parameters that can be tweaked on the servers to address this issue. We decided to modify the two significant ones:
- ARP timer
- ARP table size
After some amount of testing with these parameters we figured out the correct settings that would make a significant effect on the broadcasts hitting the core switches.
Based on the above finding we modified the parameters as following across our Data Centers and the results were amazing. The ARP traffic hitting the core switches reduced by 80 to 85%, significantly improving our network capacity and performance.
Best Practices (for Networks that are very static)
net.ipv4.neigh.default.gc_thresh3 = 4096 net.ipv4.neigh.default.gc_thresh2 = 2048 net.ipv4.neigh.default.gc_thresh1 = 1024 net.ipv4.neigh.default.gc_interval = 3600 net.ipv4.neigh.default.gc_stale_time = 3600 net.ipv4.neigh.eth0.gc_stale_time = 3600 net.ipv4.conf.all.arp_filter = 1 net.ipv4.conf.eth0.arp_filter = 1 net.ipv4.conf.all.arp_ignore = 1 net.ipv4.neigh.default.base_reachable_time_ms = 1200000 net.ipv4.neigh.eth0.base_reachable_time_ms = 1200000
Sometimes, we tend to concentrate more on the design and higher-level aspects while ignoring finer details that have potential to impact the performance and scalability significantly.
ARP broadcasts not only have the potential to improve network (switches) but also helps in bringing in more efficiency in terms of utilisation of server/systems.
ARP Broadcasts, if not configured in a proper way can have significant negative impact on the overall performance of the network and consequently the applications running on it. This also leads us to believe that there are tuneable parameters which can improve system/network and overall performance.