[Linux] net.core.netdev_max_backlog tuning

Summary

• 1st step

# cat /proc/net/softnet_stat

• 2nd
  check 2nd column of stdout

• 3rd
  Is all zero?? No problem!! Status is good!

• 4th
  Not 3rd?? doubling net.core.netdev_max_backlog

# sysctl net.core.netdev_max_backlog
net.core.netdev_max_backlog = 1000
# sysctl -w net.core.netdev_max_backlog=2000
net.core.netdev_max_backlog = 2000

• 5th
  Go to 1st until satisfy 3rd

The Adapter Queue

The netdev_max_backlog is a queue within the Linux kernel where traffic is stored after reception from the NIC, but before processing by the protocol stacks (IP, TCP, etc). There is one backlog queue per CPU core.
A given core's queue can grow automatically, containing a number of packets up to the maximum specified by the netdev_max_backlog setting.
The netif_receive_skb() kernel function will find the corresponding CPU for a packet, and enqueue packets in that CPU's queue. If the queue for that processor is full and already at maximum size, packets will be dropped.
To tune this setting, first determine whether the backlog needs increasing.
The /proc/net/softnet_stat file contains a counter in the 2rd column that is incremented when the netdev backlog queue overflows. If this value is incrementing over time, then netdev_max_backlog needs to be increased.

Each line of the softnet_stat file represents a CPU core starting from CPU0:
Line 1 = CPU0
Line 2 = CPU1
Line 3 = CPU2
and so on. The following system has 12 CPU cores:

# wc -l /proc/net/softnet_stat
12

When a packet is unable to be placed into a backlog queue, the following code is executed where get_cpu_var identifies the appropriate processor queue:

 __get_cpu_var(netdev_rx_stat).dropped++;

The above code then increments the dropped statistic for that queue.
Each line in the softnet_stat file represents the netif_rx_stats structure for that CPU.
That data structure contains:

struct netif_rx_stats
{
 unsigned total;
 unsigned dropped;
 unsigned time_squeeze;
 unsigned cpu_collision;
 unsigned received_rps;
};

The 1st column is the number of frames received by the interrupt handler.
The 2nd column is the number of frames dropped due to netdev_max_backlog being exceeded.
The 3rd column is the number of times ksoftirqd ran out of netdev_budget or CPU time when there was still work to be done.
The other columns may vary depending on the version Red Hat Enterprise Linux.
Using the following example, the following counters for CPU0 and CPU1 are the first two lines:

# cat softnet_stat
0073d76b 00000000 000049ae 00000000 00000000 00000000 00000000 00000000 00000000 00000000
000000d2 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0000015c 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0000002a 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
...

For the above example, netdev_max_backlog does not need to be changed as the number of drops has remained at 0:

For CPU0
Total dropped no_budget lock_contention
0073d76b 00000000 000049ae 00000000
For CPU1
Total dropped no_budget lock_contention
000000d2 00000000 00000000 00000000

The statistics in each column are provided in hexadecimal.
The default netdev_max_backlog value is 1000.
However, this may not be enough for multiple interfaces operating at 1Gbps, or even a single interface at 10Gbps. Try doubling this value and observing the /proc/net/softnet_stat file.
If doubling the value reduces the rate at which drops increment, double again and test again. Repeat this process until the optimum size is established and drops do not increment.
The backlog can be changed with with following command, where X is the desired value to be set:

# sysctl -w net.core.netdev_max_backlog=X

Reference : Red Hat Enterprise Linux Network Performance Tuning Guide | Bainbridge, Maxwell