Error NoFreeWorkers

[temporalio]

Error NoFreeWorkers

[wolfy-j]

Sounds like you are over-saturating your worker. In PHP SDK you can set the worker option withMaxConcurrentActivityExecutionSize to limit the number of jobs assigned to the worker at the moment of time. Set it to the number of workers you have. If your task is not IO heavy you can set it to a higher value.

You can control this value per task queue, so make sure that if you consume multiple task queues the sum of concurrent activity executions does not exceed the number of available workers.

$worker = $factory->newWorker(
    'taskQueue', 
    \Temporal\Worker\WorkerOptions::new()->withMaxConcurrentActivityExecutionSize(10)
);

[rustatian]

@changwuf31 There is a process per worker and the root process for the rr-temporal. For the temporal no extra goroutines. But such methods like GetFreeWorkers and Make Request are not atomic. This is not suitable for the temporal, but for the http plugin. Where you can request info for the free workers, and some other service can make a request before you make it. For the temporal plugin, @wolfy-j suggestion is totally valid, we will cover that case in the documentation and print warning message if you allocate a mismatched number of the rr-temporal workers (activities) and MaxConcurrentActivityExecutionSize.

[rustatian]

You can increase the number of workers in the config or increase max_allocation_time if that suitable.

[changwuf31]

I see, yes, we can change the allocate_timeout, but in our case, we're running this in the kubernetes environment.

Let's say we have pod A and pod B, each has 5 workers,
there are 10 jobs on the temporal server,
and each job require 5 minutes to complete.
So if pod A acquire 7 jobs, and pod B acquire 3 jobs,
then pod A will experience ErrNoFreeWorkers, meanwhile pod B still has free workers.

My suggestion is for a pod to only acquire jobs if there are available workers, I wonder if it is possible ?
We'd prefer for this job to only be retried if the execution failed, not because of no free workers available.

Thank You & Best Regards,

[rustatian]

@changwuf31 Got you. At the moment there is no API to check if there are free workers in the pool except for the Workers RPC call which shows workers status to the console.
This is also easy to implement, but this is not an atomic request. For example:

1. You make a request. 2 free workers in the pool.
2. You assume there are 2 workers and make a request.
3. Between p.1 and p.2 different service made a request and allocate these 2 free workers.
4. You make a request keeping in mind, that there are 2 workers and request failed.

[wolfy-j]

I'll add this information to the documentation.

[changwuf31]

@changwuf31 Got you. At the moment there is no API to check if there are free workers in the pool except for the Workers RPC call which shows workers status to the console.
This is also easy to implement, but this is not an atomic request. For example:

1. You make a request. 2 free workers in the pool.

2. You assume there are 2 workers and make a request.

3. Between p.1 and p.2 different service made a request and allocate these 2 free workers.

4. You make a request keeping in mind, that there are 2 workers and request failed.

I assume there is only 1 service that make the request, am I mistaken ?
On one pod, there is only one roadrunner-temporal process, or is there a few goroutine that make the request to the temporal server ?
If all workers and workflows operate on one namespace, is this still an issue ?

[rustatian]

@changwuf31 I guess that I can close this issue since we provided advice on how to work with such behavior. Can I?

[changwuf31]

@48d90782 yes, please close it, there is another issue, we're not sure if it is related or not, but we can re-open this later, if it IS related