Run drush commands in parallel

[jigarius]

Run drush commands in parallel

[jigarius]

Some more thoughts for making this commonly-requested feature a reality:

• When a command is being executed,
  • Collect the resulting output.
  • Ensure that no other Drall sub-instance is printing output at the moment.
  • Print the collected output.
  • Preferably, this should only be used when running in parallel mode 🤔
• When a command is issued in parallel mode,
  • Determine the number of parallel processes requested, say, p.
  • Determine the sites on which the commands have to be executed.
  • Break down the list of sites into p number of sets.
  • Issue one drall command for each of these sets in non-parallel mode.

Progress tracking

This can probably be dealt with in a separate issue where Drall shows better progress with progress bar(s). Or maybe provide reporting formats like progress, text, dots, etc.

• Option 1: Each process tracks its own progress.
• Option 2: The main drall command (that's running in parallel mode) tracks the progress.

For this issue, we just keep things simple and not think about tracking progress.

[jigarius]

Instead of running n drush commands with sub-lists of the sites, it seems that using a queue/workers approach will be better. I've updated the issue description to show the new proposed solution. Let's see how it goes.

[jigarius]

Here's a sample task file. This is generated by Drush for internal use, so the user doesn't need to worry about it.

Location: /tmp/drall/UUID.drallq.json.

{
  "version": "x.y.z",
  "status": {
    "total": 5,
    "pending": 3,
    "running": 1,
    "done": 1,
  },
  "task": {
    "command": "drush --uri=@@uri core:status --fields=site",
    "placeholder": "uri",
    "todo": {
      "ID": {
        "id": "ID",
        "status": "pending"
      }
    }
  }
}

The respective lock file would be located at: /tmp/drall/UUID.dralltask.lock. If the file exists, then a lock is in place.