How to perform a task across all workers in a pool?

[epgsql]

How to perform a task across all workers in a pool?

[mdaguete]

Hello,

Implement your worker as gen_server, and in code send a message to self() with erlang:send_after, in handle_info catch the message to do the timed operation.

Regards.

[sata]

yeah, that would work for the ping scenario.

But assume a situation where I would want to invoke a command for all workers to perform a given task in a broadcast manner. What options would I have then?

[sata]

I find the need to have an additional process who owns riakc_pb_socket process a poor solution. Since it implies a lot of boiler plate to be written or introduce knowledge of 'this is not riakc_pb_socket process but a general wrapper' to whom ever checks out the worker, it's not a big issue to do this I just find it unnecessary and see it as a missing feature.

In the latest client code for Riak there is keep alive option for tcp but that's also a poor substitute if you're behind load balancers who sets up two tcp session on either side of itself.

I'm quite curious to know what paths people have taken here besides the mentioned one.

[seth]

I've implemented similar health checks by putting that code into the client (the pooled member) itself.

What's the interface you'd like to have for communicating with all pool members? Do you want to apply the same function to each member? How do you want to handle members that are busy? Or do you only want to worry about free members?

[sata]

Hi Seth

For now free members should be enough. I don't see an equally cheap way of getting out all members as free members. One way would be to take the keys from #pool.all_members dict. If you've got other suggestions I'm open to experiment with it.

call_free_members(PoolName, Fun, Timeout)

Where it applies Fun over all free members taken from #pool record and return the result.

handle_call({call_free_members, Fun}, _From, #pool{free_pids = Pids} = Pool) ->
  {reply, {ok, do_call_free_members(Fun, Pids)}, Pool};

do_call_free_members(Fun, Pids) ->
  [do_call_free_member(Fun, P) || P <- Pids].

do_call_free_member(Fun, Pid) ->
  try {ok, Fun(Pid)}
  catch
    exit:{Reason, _What} ->
      {error, Reason}
  end.

I'm just thinking what the consequence of this would be. One obvious consequence is a free member X could be performing Fun while a user takes the same member to perform a task.

Though this would be the same if I were to put such code in pooled member so I don't see this as a big problem.

I'm experimenting a bit now and I can make a PR later on if you're ok with this approach.

[seth]

That seems like a fairly reasonable approach. On the try/catch you'll want to not match on the error type since things could crash with throw, exit, error. If an approach as above would help you, I'm up for merging it with some level of tests and docs. :)

[sata]

Okay. Created PR #52.