Rate limit a queue?

[chrisboulton]

Rate limit a queue?

[danhunsaker]

In most instances, rate limiting is not desirable on a queue's execution - most use cases want the queued jobs to complete as quickly as possible. As such, this kind of functionality is not included in PHP-Resque itself.

That said, it's not very difficult to implement this in your actual job classes. If you want a certain type of job to be limited, include a short delay in the setUp() (or tearDown()) method for that job's class. If you really do want an entire queue to be limited, regardless of the types of jobs in it, you can create a job "superclass" which checks which queue the job is from and adds the delay when appropriate, then have the other job classes extend it. Just make sure you call parent::setUp() (or parent::tearDown()) if your child classes define their own setUp() (or tearDown()) method.

I'm interested in implementing something along these lines. I think it can be made generic enough to make sense in the php-resque ecosystem. I'm working with a high volume (millions of jobs per day) job queuing system that I'm looking to replace. The one feature I can't seem to find on any of the available queuing systems is the ability to limit concurrency by arbitrary values. E.g., a single client can only run X concurrently, a single type of job can only run Y concurrently, and any combination of those factors. Maybe this is too granular for php-resque? Any thoughts on this would be appreciated.

[danhunsaker]

Sounds like a good candidate for a plugin, from where I'm sitting. Indeed, the original version of Resque, written in Ruby, has numerous plugins that add just such functionality, each providing one of the potential approaches you might wish to take. I haven't seen many PHP-Resque plugins, and even if some exist, I'm fairly certain there's a large gap between the plugins for Ruby's version and those for PHP's. That said, plugins for this exact behavior may already exist for PHP-Resque as well. I'm not certain.

I do know that PHP-Resque-Scheduler, a sister project to this one (which acts a lot like a plugin, though it technically isn't one), supports the "execute at" and "execute after" types of functionality you'd want to have for a robust and efficient rate limiting approach. The approach I mentioned above could easily be read to include this functionality, but in practice I actually took the significantly less-efficient approach of simply requeuing jobs whose rate limiting period hadn't expired. This completely rearranged most of my tasks chronologically (which is fine since they are, by design, order-agnostic, but it's important to be aware of anyway), and took a hell of a lot more processing power (and network overhead, since my Redis server is on a different VM from my worker processes) than would otherwise have been necessary. If I had it to do again (and when I get a chance to refactor, I will), I would absolutely make Scheduler a part of the equation.

Anyhow, my point is that it's technically possible, but it's complex enough and in low enough demand that making it part of the core Resque library is currently, from my perspective, out of scope. I should stipulate, however, that while I have made a large number of contributions to this project, I am in no way actually responsible for it - I can't make these kinds of decisions, and I definitely have no power to accept pull requests and such. So my comments are simply feedback from my own point of view, and should be read as such.

Interested in others' points of view on this. @chrisboulton's opinion is, of course, the only truly relevant one, but he has, in the past, considered others' PoVs as well in making his own decisions about the project, so it's useful to have that input already.

Thanks for the response.

I initially considered the requeue option, but it sounds like that may not be the best route. The alternative that I came up with was having a separate queue for every possible combination of restrictions. So if you have clientId and jobType, you would have a queue for each value combination of clientId and jobType (e.g., encodeSomething:clientA, encodeSomething:clientB). Using a separate data structure to house current state of things could keep the restrictions in check.

From what I can tell, that model is way outside the scope of php-resque, but I thought it was worth mentioning.