Concise way to depend on scripts in child and dependency workspaces

[google]

Concise way to depend on scripts in child and dependency workspaces

[appsforartists]

I'm faking this in the interim by adding this to the root package.json

{
  "private": true,
  "workspaces": [
    "packages/*"
  ],
  "scripts": {
    "build": "wireit"
  },
  "wireit": {
    "build": {
      "dependencies": [
        "./packages/core:build",
        "./packages/preact:build"
      ]
    }
  },

where core and preact are the folder names of my local packages. Now I can, e.g. yarn run build watch and have all my packages transitively watched without running into serial blocking issues.

[cefn]

Maybe I misunderstood the scope of this feature, but I'd be expecting globs and optionally lists in the workspaces and dependencies fields, as I would value e.g. having different tasks for my apps vs packages, or maybe different text matches in package names. For example my current need is something like this...

{
  "wireit":{
    "test":{
      "dependencies":[
        "test:*"
      ],
    },
    "test:apps":{
       "dependencies":[
         {
           "script:"test",
           "workspaces":"apps/*"
         }
       ]
    },
    "test:packages":{
       "dependencies":[
         "test:packages:store",
         "test:packages:queue",
       ]
    },
    "test:packages:store":{
       "dependencies":[
         {
           "script:"test",
           "workspaces":"packages/store*"
         }
       ]
    },
    "test:packages:queue":{
       "dependencies":[
         {
           "script:"test",
           "workspaces":"packages/queue*"
         }
       ]
    },
  }
}

Currently I'm having to automate the composition of an array like this to give me parallelism which is pretty unpleasant (and fragile) compared to being able to glob it based on an assumed-consistent structure. I haven't bothered to create aliases either (e.g. apps vs packages) as it's too complicated to keep all the lists complete. Typically the apps have MUCH more complex tests and longer runtimes, so I'll often focus on package (unit) tests for rapid TDD feedback, even though the apps DO depend on the packages. Aliasing apps vs packages would mean I can suppress apps in the short-term, but run them in the end. Given the current expressivity I tend to just live with serial execution, or manually watch package-specific test scripts. I'd rather this was able to be done in an aliasing layer.

{
  "wireit": {
    "test": {
      "dependencies": [
        "./apps/counter-react-ts:test",
        "./packages/queue:test",
        "./packages/store:test",
        "./packages/store-edit:test",
        "./packages/store-follow:test",
        "./packages/store-react:test"
      ]
    }
  }
}

[deebloodd]

is it possible to have glob on dependencies for example

{
  "wireit": {
    "test": {
      "dependencies": [
        "./packages/*:test"
      ]
    }
  }
}

I think this will make it less magic and easier to understand

I think this would be really nice. I was just looking through the docs to see if you could do this. I was just looking to see if this is something I could do quickly. It avoids a lot of API design and sticks with the original syntax.

[alyahmedaly]

is it possible to have glob on dependencies
for example

{
  "wireit": {
    "test": {
      "dependencies": [
        "./packages/*:test"
      ]
    }
  }
}

I think this will make it less magic and easier to understand

[black7375]

IMO, I hope it can be set like turborepo's dependsOn.
If packages.json has workspaces or file pnpm-workspaces.yaml exists, it can be recognized as a workspace.

{
  "scripts": {
    "build": "wireit",
    "lint": "wireit",
    "test": "wiret",
    "test:all": "wiret"  // Root only
  },
  "wireit": {
    "build": {
      "dependencies": ["^build"]
    },
    "lint": {
     },
    "test": {
     "dependencies": ["build"]
    },
    "test:all": {
      "dependencies": ["lint", "test"]
    }
  },
  "workspaces": [
    "packages/foo",
    "packages/bar"
  ]
}

[rictic]

What would "^build" in a dependency mean?

[black7375]

It's a topological dependency.
The explanation is well described in turborepo link.

[aomarks]

Just to clarify, there are two related operations related to this space (originally this issue was just discussing [1], but I think it's helpful to discuss both together).

[1] Run <script> in all of my workspaces (parent → child).

In Turborepo, I believe [1] just happens automatically, so if you run turbo run build from your root, it will run build in all workspaces.

With Wireit, you can already just do npm run build -ws to achieve this, which is the standard npm workspaces approach. However it's not fully optimal, because npm doesn't parallelize. So we will also support a syntax like this:

{
  "name": "root",
  "scripts": {
    "build": "wireit"
  },
  "wireit": {
    "build": {
      "dependencies": [
        {
          "script": "build",
          "packages": "workspaces"
        }
      ]
    }
  },
  "workspaces": [
    "packages/foo",
    "packages/bar"
  ]
}

Which is equivalent to:

{
  "name": "root",
  "scripts": {
    "build": "wireit"
  },
  "wireit": {
    "build": {
      "dependencies": {
        [
          "./packages/foo:build",
          "./packages/bar:build"
        ]
      }
    }
  }
}

[2] Run <script> in all of my dependencies (child → siblings).

This is what the ^build syntax in Turborepo does.

In Wireit I think we should support it with a similar explicit syntax to [1]:

{
  "name": "foo",
  "scripts": {
    "build": "wireit"
  },
  "wireit": {
    "build": {
      "dependencies": [
        {
          "script": "build",
          "packages": "dependencies"
        }
      ]
    }
  },
  "dependencies": {
    "bar",
    "baz"
  }
}

Which is equivalent to:

{
  "name": "foo",
  "scripts": {
    "build": "wireit"
  },
  "wireit": {
    "build": {
      "dependencies": [
        "../bar:build",
        "../baz:build"
      ]
    }
  }
}

[appsforartists]

Mild bikeshedding, but dependencies can have both modules in your control (my-other-package, perhaps populated with a symlink by yarn link my-other-package) and ones downloaded from npm (e.g. tslib). Curious to see how you handle that.

Would "packages": "dependencies" only follow symlinks?

[aomarks]

Mild bikeshedding, but dependencies can have both modules in your control (my-other-package, perhaps populated with a symlink by yarn link my-other-package) and ones downloaded from npm (e.g. tslib). Curious to see how you handle that.

Would "packages": "dependencies" only follow symlinks?

Great question! I think there is indeed an important distinction between sibling workspace dependencies, and npm link'd packages. I think we only want to include the former.

I think the right solution is probably to find your workspace root, grab the set of all workspaces, and then filter them by the dependencies of the starting package. Or in other words, it's the intersection of your dependencies and your sibling workspaces.

[black7375]

My first suggestion was that I would like to be implicit syntax at workspace level.

{
  "scripts": {
    "build": "wireit"
  },
  "wireit": {
    "build": {
      "dependencies": [
        {
          "script": "build",
          "workspaces": true
        }
      ]
    }
  },
  "workspaces": [
    "packages/foo",
    "packages/bar"
  ]
}

to

{
  "scripts": {
    "build": "wireit"
  },
  "wireit": {
    "build": {
    }
  },
  "workspaces": [
    "packages/foo",
    "packages/bar"
  ]
}

---

The explicit syntax is easy to implement and may have advantages in the performance, but it becomes too complicated.
For example, if you convert the first proposal to an explicit syntax,

{
  "scripts": {
    "build": "wireit",
    "lint": "wireit",
    "test": "wiret",
    "test:all": "wiret"
  },
  "wireit": {
    "build": {
      "dependencies": ["^build"]
    },
    "lint": {
     },
    "test": {
     "dependencies": ["build"]
    },
    "test:all": {
      "dependencies": ["lint", "test"]
    }
  },
  "workspaces": [
    "packages/foo",
    "packages/bar"
  ]
}

It is inconvenient because it is too long and needs to set various files.
I think it is good to dependencies build at the top level in terms of cohesion.

{
  "scripts": {
    "build": "wireit",
    "lint": "wireit",
    "test": "wiret",
    "test:all": "wiret"
  },
  "wireit": {
    "build":  {
      "dependencies": [
        {
          "script": "build",
          "packages": "workspaces"
        }
      ]
    },
    "lint": {
      "dependencies": [
        {
          "script": "lint",
          "packages": "workspaces"
        }
      ]
     },
    "test": {
      "dependencies": [
        "build",
        {
          "script": "test",
          "packages": "workspaces"
        }
      ]
    },
    "test:all": {
      "dependencies": ["lint", "test"]
    }
  },
  "workspaces": [
    "packages/foo",
    "packages/bar"
  ]
}

{
  "name": "foo",
  "scripts": {
    "build": "wireit"
  },
  "wireit": {
    "build": {
      "dependencies": [
        {
          "script": "build",
          "packages": "dependencies"
        }
      ]
    }
  },
  "dependencies": {
    "bar",
    "baz"
  }
}

Not only turborepo but also yarn manage it at the workspace level.

[boeckMt]

is it possible to have glob on dependencies
for example
...

Maybe this could be complicated, if not all packages from packages/* are dependencies for the script.
E.g. package lib2 depends on lib3, lib4 and lib5

"workspaces": [
    "packages\\libs\\lib1",
    "packages\\libs\\lib2",
    "packages\\libs\\lib3",
    "packages\\libs\\lib4",
    "packages\\libs\\lib5",
   ...
  ]

{
  "name": "lib2",
  "dependencies": {
    "lib3": "*",
    "lib4": "*",
    "lib5": "*"
  },
  ...
  "wireit": {
    "test": {
      "dependencies": [
        "./packages/libs/*:build"
      ]
    }
  }
}

then we would have something like

"wireit": {
    "test": {
      "dependencies": [
        "./packages/libs/*(lib3|lib4|lib5):build"

        or

        "./packages/libs/!(lib1|lib2):build"
      ]
    }
  }

If all dependencies should be listed implicitly, I like the suggestion of #491

#491 Maybe we could introduce a "command" like wireit detect-dependencies...