fbzmq threading model

[facebookarchive]

fbzmq threading model

[saifhhasan]

Hello,

ZMQ internally has multiple IO threads (configurable when you create context). These threads are responsible for actually performing IO. Socket read/recv are just commands exchanged between ZMQ socket interface and actual IO threads (often there is 1 IO thread as it can serve upto several Gbps of throughput as per ZMQ's doc).

Give a shot to what you intend to do and ZMQ's architecture should be able to handle it with scale you intend.

Once message is received you can pass it to worker thread or do processing yourself. In former case you will need method to interact between threads.

NOTE: fbzmq is just a thin wrapper over the ZeroMQ library in C, with some additional constructs that makes programming in C++ easier and less error prone with ZMQ.

[kreuzerkrieg]

Thanks for response. My question is, when and how I return reply from the server after the message is processed in separate worker thread. Consider following code:

addSocket(fbzmq::RawZmqSocketPtr{*serverSocket}, ZMQ_POLLIN, [this](int) noexcept {
           auto msg = serverSocket.recvOne(serverConfiguration.readTimeout);
           auto resp = requestProcessor(msg);
           serverSocket.sendOne(resp.value());
       });

It is a simple construct which blocks the processing thread, message received --> processed --> reply sent. Now lets make the processing asynchronous

addSocket(fbzmq::RawZmqSocketPtr{*serverSocket}, ZMQ_POLLIN, [this](int) noexcept {
           auto msg = serverSocket.recvOne(serverConfiguration.readTimeout);
           threadPool.AddTask([self{shared_from_this()}](auto&& msg){self->requestProcessor(msg);});
           serverSocket.sendOne(resp.value()); // <-- now what?
       });

In this case the processing is done asynchronously, but how I send the reply? Passing the socket to the thread pool to signal the completion sounds like a bad idea since (AFAIR) zmq sockets are not thread-safe and cannot be used simultaneously from different threads.

[saifhhasan]

You add socket to event loop ... There are two ways to send reply I can think of

• When you receive an event on socket and send response to the same socket or other
• Or timeout that occurs in loop (EventLoop supports scheduling timeouts), which will invoke callback function and it can send message to socket.

One thing to note here is that, if callback is invoked that means message is always ready to read so you can read with NON_BLOCKING flag to avoid getting blocked.

I think what you're asking here is general async programming question. You might like to refer to some guides available online to achieve that.

[kreuzerkrieg]

@saifhhasan, I dont think this question is in any way related to async programming. It is a question of zmq patterns to employ in such a case. For example, the answer may be Paranoid Pirate Pattern which is described in the zmq guide.