Dynamic Workers and Queues for scaling?

[koudelka]

Dynamic Workers and Queues for scaling?

[koudelka]

Hm, I'm able to start workers on new nodes. What you're describing sounds like you might be inadvertently trying to start two sets of workers on the same node for the same queue. Are you sure you're running Honeydew.start_workers/3 on two separate nodes?

Did you check out the "global" example? I just tried it with two different "worker" nodes, seemed to work fine.

One thing you can't do is dynamically change the number of workers, on any one node, servicing a queue. Well, you can, but you have to stop the workers first, change the :num argument to Honeydew.start_workers/3, and start them up again. I'd love to implement real auto-scaling, but I think that's a can of worms when you get into it.

It's not documented (my bad), but the num argument actually indicates the number of workers that should be started per queue node in the cluster. So if you specify num: 5, and then have two queue nodes, you'll end up with ten workers running on each of your worker nodes. When a new queue node joins the cluster, you'll get another five. If a queue node becomes uncontactable, the number reduces by five, etc..

I haven't messed around too much with the particulars of adding new nodes dynamically to the Mnesia queue, I think there's a little bit of trickiness around that, @aaronrenner did a bit of work on that recently, he might be able to fill you in.

I'm more than happy to take a look at a minimal reproduction, if this doesn't help.

[DeLongShot]

@koudelka Thank you for your answer. You were spot-on. I realized that I had inadvertently started workers on the same node for the same queue.

Yeah, dynamically changing the number of workers would be great, but I understand that problem could get very complicated quickly. I figured that I could stop and start them again if I really need that.

Thanks for the heads up about the num argument. I may mess around with adding new queue nodes dynamically, but I think I've come up with a good compromise for now.

I'll close this issue as there's no bug associated with it and these features would be too complicated to add for now. Thanks again for the quick answer.

[koudelka]

do let me know if you run into any other snags :)