victorpoluceno/mrchunks

mrchunks's Issues

Fix non deterministic scheduling

It seems that python is reusing memory address when a functions is sent as parameter to scheduler. Because of that our scheduler just re-schedule same functions as before changing position on deque structure.

Make test_concurrent a real unittest

Invevestigate if zmq pooler is good for performance

Right now we are using zmq pooler to make zmq sockets non blocking and cooperative. Need to investigate is this is a good idea for performance reasons or if we should look into zmq ioloop.

Add options to configure a mailbox as localhost only by default or listen on 0.0.0.0

Investigate if arbiter should be a singleton or we should put the engine number on the pid

Right now, if we start two engine at same time, the pids will collide. So we can make arbiter a singleton or make it put a number on the pid to make it unique on that arbiter.

Investigate a zmq request reply broker using router

This would have the benefit of not needing one socket per process, what seems really bad. Process need to be as small as possible. With a zmq broker we may need one socket connection per node, and this node would route messages using zmq router. Don't now right if we can use zmq router as inter processing route.

Implement resolver algorithm

Right now our resolver algorithms is a joke. It's need to work like this:

If address is a integer, it is a pid, in this case it should communicate locally, may be using inproc sockets from zmq.

If it's a tuple, it's a address/port, so is not locally. Should use a normal tcp socket from zmq.

Mailbox must show when a incoming connect happens

Think about a design for how port should be allocated

Investigate if we can turn concurrent into a concurrent.futures interface

Python has a very elegant futures interface with implementations for pools of threads or process. May be we can implement or concurrent module as a future/pool interface like Python stdlib does.

Investigate how to make zmq cooperate with our scheduler

Right now when zmq is sending/receiven on a socket it will block our scheduler, need to make it cooperate. One alternative is to use gevent as our scheduler and zmq green, but we need to check if gevent will work with multuprocessing without monkey patch.

Find whta is best serializer for a streaming workload