Is there any way to deliver large size packet to FPGA without DPDK?

[corundum]

Is there any way to deliver large size packet to FPGA without DPDK?

[alexforencich]

I think it depends a bit on exactly what you want to do. One option is to simply increase the configured MTU size, and then the kernel will provide larger frames. Is this for networking applications, say to implement TSO or LSO? In that case, I think you can just advertise support for LSO in the driver, and the kernel will provide larger frames. However, there are currently a few limitations in the hardware that might be a problem. First is descriptor support. Large frames generally mean a longer scatter/gather list, and right now hardware support is limited to 8 pointers per packet. Variable-length descriptor support should relax this and permit larger descriptors if necessary. Second issue is how the transmit engine works. All packets are fully written into an internal buffer. The buffer must be large enough to store a handful of packets in order to achieve reasonable levels of performance, so if you're trying to transfer very large packets then you'll either have to burn a lot of block RAM for the buffer, or eat the performance hit of storing fewer packets in the buffer. It's also possible to change the design of the datapath to alleviate some of these issues, such as by reading packet data out of the buffer before the entire packet has been written into the buffer, but this would require some more involved changes.

[ParkGyuhwan]

Thank you for your detailed answer. I will try to get a big size packet based on your answer later, and yes, I'm trying to implement TSO, and furthermore USO. However, it is difficult to find a way to modify the codes in order to implement TSO. Do you have any good idea for us? Now I'm just trying to check the packet that is received by tx_checksum and segment this packet before transmit it.

[alexforencich]

Unfortunately, corundum isn't really designed to support TSO at the moment. There are probably going to be significant architectural changes required to make that work, and some of these changes may result in unacceptable trade-offs to things like transmit scheduling and TDMA performance. It's definitely something that I am open to considering, but the last time I looked at how to implement something like that, I didn't see a clear way to make it work without some serious trade-offs in other areas. I will note that the transmit and receive engines in Corundum are rather simple, there are probably all sorts of ways that they could be improved to make them more flexible. Possibly there is another architecture that can support both TDMA and LSO, but I have not had an opportunity to sit down and see if I can hammer out something better. Perhaps it is time to revisit this.

Corundum was originally designed to support research in optically switched networks, and as a result the transmit scheduling and TDMA capabilities are vital as well as multiple ports and transmit schedulers on the same interface, and the compromise at least for the time being is scatter gather + software GSO.

[alexforencich]

Also, I should mention that if this is interesting enough for you, I would be willing to perhaps have some sort of a brainstorming session to see if there is a reasonable way to make something like that work. Some sort of LSO capability in corundum would certainly be quite useful.

[alexforencich]

So I put some thought in to this, and I actually have an idea about how LSO might be implemented in Corundum. In the original version of Corundum, the only thing that actually saw the descriptors was the transmit pipelines, and that's a problem because that means the entire LSO packet would have to be sent out of the same port at the same time, which is not so great. However, after reworking the DMA subsystem, I pulled out the descriptor read component as its own module, so now all of the descriptors are read in one place, and the descriptors themselves are handed off to the transmit pipeline (instead of only the descriptor address). This was done for a couple of reasons, partially to simplify the internal datapath and partially as a step towards being able to support variable-length descriptors where the descriptor has to be at least partially parsed to determine the address of the next descriptor.

Anyway, with this centralized handling of the descriptors, I think it would be possible to implement LSO through descriptor rewriting. Here's the idea: when an LSO descriptor is read in by the descriptor read module, instead of returning the descriptor verbatim, the module will return an edited descriptor that corresponds to the first piece, and then it will store the offset in the queue state. The next request for a descriptor for that queue will get the next piece, etc. In this way, the queue state only needs to hold the current offset into the payload, and if the descriptor gets kicked out of the cache before all of the pieces have been processed, it can simply be re-read from host memory. Actually, maybe rewriting is not necessary, maybe all that is needed is some metadata to indicate to the transmit engine which part of the packet needs to be sent, and then the transmit engine can ignore the rest of the payload. Either way, the idea would be to handle this at the descriptor level so that each piece can be handled separately.

This will require variable-length descriptor support (which I am currently gearing up to work on) and probably descriptor-inline headers (which will require some thought wrt. implementation). But the bottom line is that it looks like the architecture is moving in a direction where this will not only be possible, but it should be possible to implement this without any serious trade-offs.