Possible performance issues with central queues?

[iamsubingyawali]

Possible performance issues with central queues?

[Allgoodnamesargone]

Same concerns here.
Even thought using redis speed-up the queue, when you imagine of having 1k tenants that they have thousand queues each, things will not work as it would in a non multi-tenancy project/website.

Let's make an example possible case where there are 1k tenants that each has 50 queue items which is not far from real world project. There will 50k queue items in the redis db and one queue worker! of course we can make the queue worker to execute multiple queue items at the same time, but how much FR? 1k concurrents? And it is still a big mess regarding queue orders per tenant.

The best and the implementation that a multi-tenancy package/project needs to have, in my opinion, is queue and queue processor per tenant, dedicated for each tenant. not shared. however, still with one worker process to keep the server's resources usage as low as possible, not one process per tenant.

While i was writing this and thinking more and more on this, i found out that the current implementation might be the best and the only possible way while there are known issues too!

[stancl]

I don't think this is fundamentally different from single-tenant apps in any way. It's the same problem of queue scaling.

It doesn't matter if you have 50 jobs from 1k tenants, or if you have one app that dispatches 50 000 jobs.

I think the solution is to just add more queue workers. I don't think queue per tenant is a good idea because in your example you'd have to manage 1000 queue workers that are fully coupled to the tenant and would need to be managed as such — create a worker when a tenant is added, delete a worker when a tenant is removed etc.

It's probably best to treat this the same as a single-tenant app — just add more queue workers as your queue volume grows.