Very slow jobs perform method

[mojolicious]

Very slow jobs perform method

[kraih]

There's a Redis backend? 😃 Anyway, i don't see anything wrong with the perform method, the fork() and waitpid() calls certainly cost a bit of performance, but that's not a bug. Or am i missing something?

[kraih]

If you have an idea for speeding up the perform method, please open a new issue or pull request.

[alexvaluyskiy]

So, why you are using forking for task performing? For right exception handling or for good isolation or for some non-blocking features?

[alexvaluyskiy]

For example, I have 1 millions jobs (dummy jobs without any logic) and only one worker and I'm trying to perform these jobs
with forking: 382 jobs per minute
without forking: 60000 jobs per minute

that's why can't see a reason, why I should use forking version of perform method

[kraih]

All of those reasons, and don't forget that pretty much every non-trivial Perl application leaks memory, which is also the main reason Hypnotoad workers get restarted regularly. I believe it is the right default. If your jobs run so fast that you can perform 60000 per minute, why would you want to use a job queue in the first place?

But of course there are different architectures for workers, and you could handle multiple jobs concurrently with a UNIX specific prefork worker. That's why the public Minion API allows you to just ignore the perform method and build your own version of it. In fact, i'm looking forward to more pluggable backends (Minion::Backend::Redis) and worker commands (Minion::Command::minion::fast_prefork_worker) on CPAN. 😄

[alexvaluyskiy]

Maybe you are right. My point is Minion has a very good API and I can use it not only for heavy jobs, but also for some another tasks, like notifications (delayed tasks) and etc. That's why I want to have much faster worker runner for these cases.

i'm looking forward to more pluggable backends (Minion::Backend::Redis)

Right now my Redis backend passes all standart Minion tests for backends. But I should to speed up some methods. And I'm not sure about Redis transactions, which completely different from Postgres transaction. I need some tests for these cases.

[schandrakandi]

@alexvaluyskiy did you find any reliable solution to make this works ? could you please share if you have any suggestion to make the Job running faster.