Execute jobs via php-fpm rather than forking

[chrisboulton]

Execute jobs via php-fpm rather than forking

[chrisboulton]

Ok, this is seriously cool. 😎

I'd definitely be interested in having this built in, and it might help others such as @salimane who have been trying to work with the overhead of a forking model. It will also improve the error handling/catching for failed jobs as you'll be able to get the actual error message for a job that fatal errors out and would normally leave you with a "DirtyExit" exception.

It'd be super cool to get this working via sockets at some stage as well.

I can sort of see there being different strategies for job execution - Resque_JobStrategy_Fork, Resque_JobStrategy_Fastcgi, and you can configure one at runtime. Thoughts?

[salimane]

hi
@ebernhardson @chrisboulton this is interesting. one quick problem I could see is long running jobs over fastcgi. May be something could be done to adjust for this. But as long as this is space and resource efficient than fork, I'm all for it.
Thanks

[ebernhardson]

Job strategies sound like a good approach to this. I took a stab at refactoring the current code into a strategy pattern, let me know what you think. I'm not sure that perform is the right name for the method on Resque_JobStrategy_Interface, suggestions welcome.

I also have some worry about Resque_Worker::killChild, in that the InProcess strategy has no way to kill the job(when triggered from SIGUSR1), and closing the fpm socket in the Fastcgi strategy wont necessarily stop fpm from finishing the job either. Perhaps a few notes in the documentation is all that is required though.

ebernhardson/php-resque@36f5353

@salimane
Long running tasks are certainly a valid concern. We can always set max_execution_time to 0 from the worker code so it matches the CLI environment, but running some specific testing will be worth the time invested. The time and space efficiency really need to be measured to have any certainty. My intuition tells me it wont be as clear cut as the initial benchmarks show, and will depend on the use case.

[salimane]

@ebernhardson

I quickly went through the changes at ebernhardson/php-resque@36f5353 . If I understand this correctly, this proposal comes down to instead of running the jobs within a child of the worker process, run the jobs directly in the worker process. is this so ?

[ebernhardson]

@salimane The changes in the fork dont represent fcgi yet. The first step here was just to see if @chrisboulton is ok with the direction of the refactor before i start plugging in and writing tests for fastcgi. The linked refactor passes all the original tests and should have exactly the same functionality as the current php-resque code. With the fastcgi in all jobs will be executed by the php-fpm manager, similar to how nginx would run run webpages through php-fpm. Connecting and running scripts through fpm is demonstrated from the gist in the first post.

[salimane]

@ebernhardson ok got it :)

[danhunsaker]

So... How's this coming along? I see PR #81, but it doesn't have any recent updates, either...