Implement fine-grained parallelization

[flyingcircusio]

Implement fine-grained parallelization

[gforcada]

A bit unrelated, but around the parallelism topic: it would be great/needed to have a way to limit the amount of parallelism.

Scenario: I'm deploying some new changes to our Plone based website where we have 9 workers. If all 9 workers at the same time run buildout and finally the process, via supervisor, is restarted, it might very well be that some requests will be refused due to the lack of servers available, as HAProxy will not see any worker available roughly at the same time.

If instead, we could limit the parallelism to, say 3 workers, then the deployment will still be 3x faster than batou 1.x already, but still allow the website the be responsive.

[ctheune]

Yes, thanks for that! I forgot to add it as a separate ticket. I have a plan in my head already:

• for the moment I want to get the mechanism for paralellization ready
• i'm expecting to collect 2-3 projects that want/need to use it, and then
• gather the specific requirements how we want to limit things based on the actual requirements from the projects

I have a couple of things in my head how to provide syntax for those limits but I really want to base it on actual use cases ...

[gforcada]

@ctheune where is the form to sign up? 🙋‍♂️ 😄

[ctheune]

He, sorry @gforcada that I missed your signup ... there's an incoming PR from @wosc's colleagues that will allow setting parallelization limits similar to '-j' on make. That can be turned to "1" for the old behaviour. I'm also finishing up the general Python 3 migration and then I can take on more projects ... that want to migrate and help those :)