Fix and improve parallel execution

[jigarius]

Fix and improve parallel execution

[jigarius]

Thanks for the report! I'll take a look at it as soon as I have time.

[golddragon007]

My env is technically LAMP with native drush/composer + docker for rare services like solr.

[jigarius]

This might not be platform-specific. I finished the parallel execution feature in a hurry, so maybe it doesn't work correctly. I'll probably have time to check it after Aug 07.

[jigarius]

Cool! I tested the amphp/parallel library and it seems to be working like a charm. So, the next action steps for me:

• Plan the implementation of amphp/parallel
• Use Amphp libraries for parallel execution
• Invite people on this thread to try out the implementation
• Remove code related to the old parallel execution mechanism

[lems3]

Feel free to ask us to test anything anytime, we're still "evaluating" Drall, as in "it's not a critical part of our current workflow so we can install a dev version anytime and we're not going to block any work". Although we're heavily using it instead of creating new Robo/Acquia BLT code to handle tasks 😄

[lems3]

Same experience here. We run our local development stack on lando, on the Drupal 8 recipe.

[lems3]

In the meantime if there's anything that could make sense to test to try to pinpoint the issue feel free to ask 😄

[jigarius]

I had implemented the experimental version of parallel execution in a big hurry because I wanted to present it at an event. However, it seems it might be better to give it some thought and go for a more elegant, long-term solution.

Having done some research, I found 2 ways to implement a better concurrent execution mechanism.

PHP Parellel

This requires installation with PECL and appears to be the official PHP package for parallel execution. However, this requires the user to have sufficient access to install this package on the server.

amphp/parallel

This package, on the other hand, is installable via Composer, so I'm leaning more towards this one. I haven't tried it out yet, but phpro/grumphp seems to be relying on it as well.

Next steps

• Try amphp/parallel and see how it works.

[jigarius]

@lems3 sure thing. Since you mentioned Robo and Tasks, I'd like to invite you to check out Phake.

[jigarius]

@lems3 while I try to figure out the amphp/parallel and amphp/process libraries, I made a small change to the existing the parallel execution mechanism.

Will you be able to test this change out once and see if it works any better?

• Branch dev/better-parallel-execution-v1 contains a fix to the homegrown solution that is implemented currently.
• Branch dev/better-parallel-execution-v2 contains a work-in-progress usage of amphp/process.
  • The only thing pending is to figure out how to cap the number of processes to --drall-workers.

[jigarius]

I did some tests on a 200 site installation. Here's what I got:

Drall homegrown - 16 workers

real	1m35.941s
user	0m0.050s
sys	0m0.063s

Drall amphp/process - unknown workers

real	1m25.289s
user	1m5.401s
sys	0m45.819s

Thoughts

• I could spend some time to improve the way the homegrown solution works, but I don't like reinventing the wheel.
  • I'd rather use amphp/process so that I won't have to maintain that part.
• amphp/process works well and the only part I can't figure out is limiting the number of workers
  • Maybe I'll drop the --drall-workers and let amphp/process decide what's best?

[jigarius]

Thanks to the kind maintainers of the amphp/* libraries, I was able to implement parallel execution using the amphp/sync and amphp/process libraries.

Here's how you can test the work:

composer require jigarius/drall:dev-dev/better-parallel-execution-v2

Also, --drall-workers can now take a value of up to 16.

[lems3]

@jigarius I had a chance this morning to test it out.

One feedback that I have : the "Current site" display should print out the command to be executed. Simply to make sure we're not typing something wrong.

First, I did a run without any workers :

user@laptop:~/Drupal$ time lando drall exec:drush @@site.local cr
Current site: @1
Current site: @2
Current site: @3
Current site: @4
Current site: @5
Current site: @6
Current site: @7
Current site: @8
Current site: @9
Current site: @10
Current site: @11
Current site: @12


real    1m19.147s
user    0m1.189s
sys     0m0.180s

Then, I played with workers, using 16 at first, then moving down to 8, 4, and 2.
16 obviously ran the commands on all 12 sites at once. 8 was more or less capping my local ressources too, so both took a similar time to execute.

4 and 2 were not using my CPU and drive at 100%, so it did took more time. But it's still better than doing them all 1 by 1.

user@laptop:~/Drupal$ time lando drall exec:drush @@site.local cr --drall-workers=16
Current site: @1
Current site: @2
Current site: @3
Current site: @4
Current site: @5
Current site: @6
Current site: @7
Current site: @8
Current site: @9
Current site: @10
Current site: @11
Current site: @12


real    0m29.282s
user    0m1.150s
sys     0m0.159s

We tested out an sql:sync, expecting it to crash, and it did. We need to configure our things properly so each site dump uses a different filename, but that's on our side to solve.

If there's other tests you'd like me to do, you can ping me.

[jigarius]

@LEMS I wonder why you get @1, @2, etc. Are those your site alias names? You can see the entire command with flags like --drall-verbose and --drall-debug – I can't remember which one shows the command, but it's one of those.

I think a more realistic time figure would be if you do lando ssh and then run drall from there. I used to use Lando a lot and one thing I remember is that running commands through lando was quite slow in comparison to running it directly after lando ssh.

To call this issue resolved, I'll try and wrap up the pull request that I'm working on. Thanks a lot for your help.

[lems3]

@jigarius I just replaced the alias names by numbers in my comment so I wouldn't list all our sites names in a public forum 😄

I'll dig around and test flags to get the output, thanks for the info!

For us, the important element was to test how much time could be saved in our actual process, and we tend to not SSH directly in lando for our commands. And, shaving off more than half the time is already a good step for us. I will take a look at getting in the container and run drall there, if it saves more time for our local development workflow it could be nice. But the workers are already a huge step forward. We continued to play with it, with rsync and other stuff, and it's always a huge benefit.

Thanks for your time on this!

[jigarius]

The issue should be resolved, thanks to Amphp. If you think the issue persists, then feel free to reopen this ticket.