Unit tests crash emulator on fresh checkout

[zeromq]

Unit tests crash emulator on fresh checkout

[kevsmith]

Dug into this a bit further and determined the code is tripping over an assert in ZeroMQ itself in zmq::mailbox_t::send on line 194:

//  This should never happen as we've already checked that command size is
//  less than PIPE_BUF.
zmq_assert (nbytes == sizeof (command_t));

I'm not at all familiar with ZeroMQ internals but it looks like some sort of fragmented send.

[kevsmith]

I managed to narrow the crash down to shutdown_stress_test. If I configure the test to create < 46 processes it succeeds the majority of the time with just a few intermittent crashes. Dropping the processes below 30 allows it to pass every time. Anything above 46 processes crashes every single time.

I sprinkled some printf() calls in zmq::mailbox_t::send() and have been able to confirm the crash is due to a short send:

nbytes: 48, sizeof(command_t): 48
Invalid argument
rc == 0 (mailbox.cpp:179)
nbytes: 32, sizeof(command_t): 48
Assertion failed: nbytes == sizeof (command_t) (mailbox.cpp:196)
make: *** [test] Abort trap

I suspect some sort of race where the rep socket is getting closed before the req socket can xmit all its data. Will continue to poke at this as time allows.

[kevsmith]

Looks like the short sends are being triggered when a socket is either closed or termed. The NIFs and background thread aren't synchronized when either of these ops occur so I think the zmq sockets are getting torn down while still in use, thus causing the short sends. I added a ErlNifCond to erlzmq_socket_t and used it to coordinate closing between erlzmq_nif_close and the worker thread. This seemed to improve stability but didn't eliminate the crashes.

[evax]

Hi Kevin,
Sorry for the late answer, I think the solution is to be found on the zeromq tuning page.
Apart from the ulimit change, here's what's stated:

0mq uses socketpairs for internal signaling and communications.
To avoid overrunning this resource, expand these buffers.
Edit the /etc/sysctl.conf file and add two lines and reboot:
    net.local.stream.sendspace=3000000
    net.local.stream.recvspace=3000000

[kevsmith]

Thanks for the pointer to the relevant sysctl entries. Making the recommended edits seems to have fixed the problem completely. I can now run back-to-back sets of unit tests without a single crash.