Support parallel tasks

[xelalexv]

Support parallel tasks

[xelalexv]

Running tasks in parallel can be advantageous when the system running dregsy has a significantly faster network connection than any of the involved source and/or target registries. In the opposite case, we may not gain much of a speed up, since the parallel tasks would compete for the slow network connection. The same may be observed if there's just one slow source and one slow target. At any rate, having the option to add parallelism to tasks is definitely a good idea.

Implementation thoughts:

• There are a number of global entities, such as authentication tokens and lister caches. Access to those needs to be properly locked when we introduce parallelism, so that for example identical auth refreshes are not done in parallel.
• It may be necessary to validate tasks to make sure there are no duplicates or target overlaps.
• ...

[sjthespian]

That would be exactly why I'm doing it -- I'm syncing to registries that are only available via. satellite links. My uplink bandwidth is roughly 10x the bandwidth of each individual registry.

I have a poor-mans working implementation of this in meantime, I'm running this in k8s so I just spin up n pods each with a single task. We don't add new registries often, so I'm not creating too much tech debt.

Validation could be tricky. While it isn't in my use case, I could see someone wanting to have parallel syncs running to the same registry with each sync being it a separate namespace.