epoll_wait causes high CPU usage

[mellanox]

epoll_wait causes high CPU usage

[rosenbaumalex]

By design VMA is tuned to push each user application thread which is calling a socket receive API and this drives the VMA engine handle all packets. The lower latency is achieved by keeping this thread spinning on the HW until an ingress packet is received and then handling it in VMA.

This 'spinning' module is the default VMA configuration. You can turn off the high polling configuration but you'll loss the very good low latency performance.

The socket 'receive' APIs which push the Rx polling include:

• recv() recvfrom(), recvmsg(), ...
• select(), epoll_wait(), poll(), ...

So if you have 6 applicaton threads which use epoll() as their main loop so there is a high chance that your'll see 600% CPU (6 cores).

[nijinboc]

HI, I found that the value m_ready_conn_cnt is not zero (sometime seems very large number) that causes this piece of code in int sockinfo_tcp::rx_wait_helper(int &poll_count, bool is_blocking)

if (!m_n_rx_pkt_ready_list_count && !m_ready_conn_cnt)
{
going_to_sleep();
unlock_tcp_con();
}
else
{
unlock_tcp_con();
return 0;
}

never going_to_sleep();

[orkmellanox]

it seems you are correct!
thanks for catching this.
the m_ready_conn_cnt was not initialized to 0 in the constructor, so it got random high numbers for non-listen (connected) sockets.

but this actually seem to affect the recv() call and not the epoll_wait() call.
am i missing something?

[nijinboc]

HI, initially I suspect that it is epoll_wait that causes the high CPU.
But after I adding some printf message in code, I found that it is the variable m_ready_conn_cnt that causes the problem.

[nijinboc]

Sorry for the misleading

[nijinboc]

But after adding initializing m_ready_conn_cnt in the constructor, seems still has the problem

[orkmellanox]

by problem you mean 600% cpu?
using gstack, who do you see spining now?

[nijinboc]

Sorry, my mistake,
After initializing the variable to 0 in constructor, does solve the problem.
Thanks a lot for the help

[orkmellanox]

thank you!
i updated the repo with the fix.

[rosenbaumalex]

OrK fixed the issue back in Oct 2015.
closing the ticket...