Pushing the Limits of Kernel Networking

Pushing the Limits of Kernel Networking

Over the past years, Red Hat team has been working really hard on the Linux Kernel Networking Services focused on improving the performance of the kernel networking data path. A lot of work has been done to improve network performance in the last year and the performance has been improved a lot as compared to Red Hat Enterprise Linux 7.1.Pushing the Limits of Kernel Networking

Dealing with kernel networking is not a piece of cake, developers need to realize when dealing with kernel networking data-paths is that the kernel has to support a multitude of functions. It includes everything from ARP to VXLAN in terms of protocol and it should be done securely. As a result developers end up needing a significant amount of time to process each packet. However, with the current speed of network devices developers aren’t normally given that much time.  On a 10 Gbps link it is possible to pass packets at a rate of 14.88 Mpps. Also you have to keep in mind that an L3 cache hit, yes that is a “hit” and not a miss, costs Red Hat something on the order of 12 nanoseconds. There ends up being several different things that can impact the scalability of systems.

One of the biggest ones for the x86 architecture is NUMA (Non-Uniform Memory Access).  Basically what NUMA represents is that there are different costs for accessing memory for certain regions of the system. To make matters worse this all assumes the system is configured correctly.  It is possible that the system could be misconfigured. We have seen systems where all memory was only populated into one socket, or even worse all of the memory is stacked into only a few channels on one socket resulting in a significant degradation of system performance. If you are sending or receiving packets from a network interface using IPv4 addresses you will likely be making one or more FIB table lookups and see the benefits of the FIB rewrite.  If you are using a network device to send or receive traffic you will see the advantages of the DMA barriers and NAPI allocation changes.  If you are doing large sends using VXLAN over a network interface without offloads supported on the network card itself you will see the benefits of xmit_more and disabling tx-nocache-copy.