SoftNIC performance and NIC offload features (scatter-gather lists and TSO)

[netsys]

SoftNIC performance and NIC offload features (scatter-gather lists and TSO)

[sangjinhan]

You are right. When one TCP flow is considered (thus only one core is utilized in the kernel space), the Linux TCP implementation processes roughly 0.5Mpps (and their ACKs). It effectively limits its throughput to 1500B * 8 * 0.5Mpps = 6Gbps. Since the overheads mostly come from per-packet cost, not per-byte cost, increasing the MTU (TSO and LRO) will immediately boost the throughput.

Also note that you would need a multi-queue vport, and distribute packets (e.g., HashLB module) across queues. Otherwise all RX packets will be forwarded to the same core. And these ACK packets will trigger TX packets to be generated on the very same core, effectively utilizing only one kernel for all flows.

In the early days of SoftNIC there were modules that implement NIC offloading functionality, such as TSO, LRO, checksum, FlowDirector, etc. Although BESS inherited most of the SoftNIC code I decided to disable the features simply because I did not have enough cycles to maintain the features. I believe a fair amount of code is still there in the kernel module side; reviving them shouldn't be terribly difficult. You are welcome to put the fragmented code pieces back together and make it work again.

I attached the old TSO/LRO code of SoftNIC.
segmentation.c.zip

[bestephe]

Thanks for the response. It seems like I do want to build a version of SoftNIC that implements at least the TSO and IP checksumming NIC offload features. Given this, I think the "segmentation.c" file that you sent along should be helpful. Thanks for sharing it. However, I have a few follow up questions/comments:

Question 1:

Although "segmentation.c" seems like it will be helpful, this only seems to be one aspect of the BESS daemon's implementation of TSO and LRO. It seems like additional changes need to be made to the SoftNIC kernel driver and the interface between the BESS daemon and the SoftNIC kernel driver. As a quick example, it seems like SNBUF_DATA may need to be changed so that the kernel driver can pass large segments to the BESS daemon. Similarly, even though it seems like there is support for TX checksumming in the kernel driver, simply enabling the "NETIF_F_IP_CSUM" flag in the kernel driver isn't working for me at the moment. Do you have a more complete snapshot of a version of both the SoftNIC kernel driver and BESS daemon that implements TSO that you would be willing to share with me? Given some of the problems I'm currently trying to solve, I think such a snapshot could be useful in helping save me some development time.

Question/Comment 2:

The reason that I'm currently trying to use SoftNIC/BESS is to evaluate changes to two things: 1) How the NIC schedules TX packets and 2) How kernel drivers and kernel-bypass applications (DPDK) interface with the NIC, .e.g., what information is communicated through transmit descriptors and how flows are mapped to queues. While SoftNIC/BESS seems to be a good platform for evaluating changes to the NIC scheduler, the implementation of the SoftNIC kernel module makes it difficult to draw any meaningful conclusions about the overheads of different implementations of the OS/NIC (and App/NIC) interface. This is because the TX interface between the kernel module and the BESS daemon is backwards from that of a physical NIC in a way that places additional computational overheads on the kernel driver and in a way that changes how the memory shared between the kernel driver and the daemon is used and laid out. At a high level, I feel that this is a mistake. I think it would be better if the SoftNIC kernel module were as close to that of a driver for a physical NIC as possible and that any extra overheads in emulating a NIC were instead incurred by the BESS daemon, not the kernel driver. More concretely, in BESS, the SoftNIC kernel module is responsible for copying packet data into SNBUFs for use by BESS as part of its xmit function. In contrast, in a physical NIC, the kernel module would only create DMA mappings and send metadata and pointers to the NIC, which is then responsible for copying packet data into NIC memory through DMAs. Since this extra overhead of memory copying in SoftNIC/BESS is incurred by the kernel driver and this implementation detail changes how memory is laid out, it is not easy to draw any conclusions from the CPU and memory overheads incurred by the kernel when it interfaces with SoftNIC/BESS. Further, as it is currently implemented, the SoftNIC/BESS kernel module breaks the byte queue limits (BQL) kernel optimization. Given these limitations, its looking to me like I may reimplement the TX side of SoftNIC (BESS VPort).

My questions about the TX implementation of the SoftNIC kernel driver are the following: First, can anyone give me some insight into why the current implementation of SoftNIC/BESS incurs the overheads of copying packets from skbs in the driver instead of in the BESS daemon? Second, has an alternate implementation approach for the SoftNIC TX interface ever been considered? Third, in what follows, I give a high level description of what I think would be a better implementation of the TX interface between the SoftNIC kernel driver and the BESS daemon. Do you see any obvious limitations, drawbacks, or problems with taking this approach instead of the current approach taken by SoftNIC?

My high-level outline of how I think the SoftNIC's kernel driver's TX interface should change to be more faithful to how kernel drivers for physical NICs are implemented is as follows:

• In the xmit function, the driver will start by mapping the skb frags into the address space of the BESS daemon
• Next, the driver will create tx descriptors to communicate both any necessary metadata and the skb fragment pointers to BESS. These descriptors will be enqueued into the llring instead of enqueueing SNBUF pointers directly like SoftNIC currently does.
• Once the BESS VPort driver reads the transmit descriptors, it will create packets by allocating memory and copying data as necessary.
• Once the BESS VPort is done with the TX descriptors, it will update some registers shared with the SoftNIC kernel driver and generate a TX interrupt for the SoftNIC kernel driver to process.
• In the SoftNIC TX interrupt handler, the kernel driver unmaps skb fragments and frees skbs that BESS has acknowledged that it is finished with.

Thanks in advance for taking the time to read this message,

~Brent