nm_open with range of queues

[cygnus2048]

nm_open with range of queues

[vmaffione]

Hi,
It would not be enough to modify nm_open or nmport_open to support such a syntax.
In the very end, netmap open requests are submitted to the kernel by means of the struct nmeq_register, and there is no way to setup that struct to specify a range of rings. Furthermore, there is no kernel support for ranges (see netmap_interp_ringid).

However, you could achieve your goal (and potentially more efficiently than using ranges) with the following scheme:

• Thread TX1 would nm_open("netmap:ens4-${N}/T") for N in [0..3], and poll 4 file descriptors
• Thread TX2 would nm_open("netmap:ens4-${N}/T") for N in [4..7], and poll 4 file descriptors
• Thread RX1 would nm_open("netmap:ens4-${N}/R") for N in [0..3], and poll 4 file descriptors
• Thread RX2 would nm_open("netmap:ens4-${N}/R") for N in [4..7], and poll 4 file descriptors

[cygnus2048]

I see. However, I guess I'm not understanding as much as I thought since what you say is not possible seems to be exactly what is done now when I open the device in "all queue" mode using no suffix as "netmap:ens4"? That is a single fd with multiple rings as well as a single ring_id and it gets access to all the queues. The descriptor has a "first ring" and "last ring" elements for TX and RX so with this in mind, how is ring_id even used?

My concern using multiple fd's is that in my benchmarking, most of the time is spent in the NIOCTXSYNC and NIOCRXSYNC ioctl() calls and now I will need to have my thread loop those calls over multiple fd's. Hopefully the total time spent will be roughly the same as in the end, it will be the same number of packets being processed, but I don't know this for a fact.

Also, is there any benefit to having a separate fd for TX and RX? I currently use a single one for all queues that the TX and RX threads both use and I have not seen any contention. In other words, using this as opposed to your suggestion above...

• Thread TX1 and RX1 would use the result of nm_open("netmap:ens4-${N}") for N in [0..3], and poll 4 file descriptors
• Thread TX2 and RX2 would use the result of nm_open("netmap:ens4-${N}") for N in [4..7], and poll 4 file descriptors

Thanks.

[vmaffione]

Maybe the misunderstanding is about the "first ring" and "last ring" fields within the userspace library (libnetmap or netmap_user.h structs. Those are just helpers that allow the programmer to write more general programs, by scanning all the rings that are bound to a given netmap file descriptor. While those fields may induce you to think that they can assume any range of values, however, only some combinations are supported by the kernel.

Things you can do:

1. open all the hardware rings (TX+RX, or TX only or RX only)
2. open only the host rings (TX+RX, or TX only or RX only)
3. open both hardware and host rings (TX+RX, or TX only or RX only)
4. open just a single ring (TX+RX, or TX only or RX only)
   See the NR_REG_* macros in sys/net/netmap.h.

Having different fds for different threads is useful to avoid unnecessary wakeups, and to selectively sync only a subset of the rings. If open netmap:ens4, and have multiple threads using the same fd but accessing different rings, any TX or RX interrupt will wake up all the threads polling that fd, even if they only care about TX or RX wakeups.
Also keep in mind that by using poll() a single thread can sync multiple file descriptors with a single system call.

[cygnus2048]

Sorry, I forgot to reply, but thanks for your explanation and information. I was able to open fd's for each TX and RX queue and just have my threads loop over them when I have less cores than queues. Thanks again. Closing this.