bchociej/bobbin

easily spool up thread-like worker processes in node with bobbin

bobbin's Issues

it'd be nice to be able to kill the workers sometime

distinguish between 'kill when work is done' and 'kill now'.

handle repetitive/long-running work

this will require some thought. how to stop these once started, for example.

repetitive:

pool.run (cb) ->
    i = 0
    setInterval( -> cb(i++) , 1000)

long-running:

pool.run -> doWorkForever()

pool.run() should return some kind of handle to the worker/task being run

so it can be managed/killed/whatever

workers and pools should share a common, public interface

run, kill, etc are really applicable to both workers and the pool as a whole. formalize this relationship in code.

main file index.js doesn't expose module.exports, so require('bobbin') is broken

catch exceptions in workers, rethrow in main

let end user access pool workers directly

why not let the end user, e.g., grab some worker and feed it work, if that's what they wish to do. or kill it. or whatever! or push workers onto a pool. who cares?

blocked by #18

Document API in README.md

thoughts on next-level concurrency management

• limit number of tasks per worker?
  • queue them if full? on worker or master?
  • reject them if full? i.e. force onto other workers? if all workers full, queue on master?
• ability to stop/kill tasks?
• task priority?
  • per worker? globally?
• ???

API method to resize worker pool (immediate/delayed like kill())

switch to child_process and pipes for performance

cluster.fork() implies synchronous IPC for passing messages. perhaps using child_process and pipes would speed things up in certain cases.

high level data/control flow constructs

basically I want to plug caolan/async into this (or just mimic functionality as needed) to make data processing magically SMP-enabled:

async_bobbin.each(collection, function(item, callback) {
    // this iterator runs on the workers!
    callback(doSomething(item));
});

i think all or almost all of caolan/async could be adapted onto bobbin. (imagine async.auto applied to workers!) some kind of bridge/adapter/facade shouldn't be hard to make.

go native, or wrap e.g. threads_a_gogo

threads_a_gogo has done a lot of heavy lifting already to get fast threads in node. the work-sending paradigm is similar. become a nice API on top of a library such as that one? or, implement native threads ourselves?

note to self: libuv threads aren't scary at all https://nikhilm.github.io/uvbook/threads.html#core-thread-operations , neither are native addons in general http://nodejs.org/api/addons.html#addons_wrapping_c_objects

API method to run work on N (or all) workers

API methods to query size of pool, # busy workers, # dispatched jobs (per pool/worker) etc

lazy evaluation of useful constructs a la lazy.js

can lazy.js or its ilk be wrapped? seems impossible structurally. maybe the concept of sequences can be ported. it'd be cool to get the lazy evaluation mapped onto workers.

accept synchronous work

e.g.

pool.run(function() { return something; })

replaceable backend would be cool

given e.g. lambdaws, it'd be swell to support other "compute backends" e.g. lambdaws, other machines on the network, etc

automagically resolve require'd pathnames by injecting module.parent.filename

currently the poor user has to inject the full path into pool.run so that the worker's require can find it

more intelligent distribution of work to workers

round robin isn't the only option of course. maybe keep a count of active tasks on each worker and prioritize accordingly? maybe expose explicit options to direct work at certain workers. maybe allow submission of task batches; could be more efficient.

API method to create single worker (w/ simplified scheduler)