ds300/lazyrepo

Caching task runner for npm/pnpm/yarn monorepos.

Warning Currently in the prototyping stage, expect breaking changes! Get help or join in development on discord.

lazyrepo is a zero-config caching task runner for npm/pnpm/yarn monorepos.

It fits right into the niche that turborepo carved out: making package.json "scripts" scale without adopting a big industrial-strength build system like nx, bazel, rush, or buck.

lazyrepo is scary fast, despite being written in TypeScript rather than some young handsome clever funny systems language.

Aside from perf, lazyrepo comes with some big quality-of-life improvements:

• A human-friendly config format.
• Concise and timely feedback to help you tweak and debug your build pipelines.
• Palpably sensible defaults.
• You don't need to learn Rust to contribute.

Trust me, the whole situation is so delightful it will make you reach for the :chefs-kiss: emoji. And then you'll realize that there is no such emoji, but you'll want it so badly that you'll draft a proposal to the Unicode Consortium to lobby for its introduction.

Installation

Install lazyrepo globally

npm install lazyrepo@alpha --global

And also as a dev dependency in the root of your repo

npm install lazyrepo@alpha --save-dev

And finally add .lazy to your .gitignore

echo "\n\n#lazyrepo\n.lazy" >> .gitignore

Basic Usage

Running Tasks

Run scripts defined in "scripts" entries using:

lazy run <script-name>

You can pass arguments to the task script after a --

lazy run test -- --runInBand

The default behavior is optimized for "test" scripts, where the order of execution matters if your packages depend on each other.

Let's say you have three packages: core, utils, and primitives. The core package depends on both utils and primitives, and they all have "test" scripts in their package.json files.

graph TD
    A[packages/core] -->|depends on| B[packages/utils]
    A -->|depends on| C[packages/primitives]

With no config, when you run lazy run test in the project root:

• The tests for utils and primitives will begin concurrently. The tests for core will only be started if both utils and primitives finish successfully.
• If you change a source file in core and run lazy run test again, only core's tests will be executed.
• If you change a source file in utils and run lazy run test again, both utils and core's tests will be executed, in that order.

Other commands

• lazy init

  Creates a config file.

• lazy clean

  Deletes all local cache data.

• lazy inherit

  In larger projects, you often end up with the same "script" entries duplicated in lots of package.json files. Keeping them in sync can be troublesome.

  lazyrepo lets you specify the command just once.

  Replace the scripts entries with lazy inherit:

   "scripts": {
  -  "test": "jest --runInBand --noCache --coverage",
  +  "test": "lazy inherit"
   }

  Then add this in your lazy config file:

   "scripts": {
     "test": {
  +    "baseCommand": "jest --runInBand --noCache --coverage"
     }
   }

  Now when you run npm test, or whatever, in one of your package directories, it will look up the actual command to run from your lazy config file and run that.

• lazy run <script> [-- <forward-args>]

  Runs the given script, forwarding any args passed after the --

  You may filter the packages that a script should run in using the --filter option, which supports glob patterns.

  e.g. to test only packages that end in -utils

  lazy run test --filter="packages/*-utils"
  

  You may force all tasks to execute by passing the --force option.

  lazy run test --force
  

Configuration

Create a file called lazy.config.js or lazy.config.json

To create a .js config file, in your project root run:

lazy init

export default {
  scripts: {
    test: {
      cache: {
        // by default we consider all files in the package directory
        inputs: ['**/*'],
        // there are no outputs
        outputs: [],
        // a test invocation depends on the input files of any upstream packages
        inheritsInputFromDependencies: true,
      },
    },
  },
}

lazyrepo's Issues

Cache TypeScript config file bundles

Follow up to #4

If the user uses a TypeScript config file, we bundle it with esbuild before loading it.

If we can get the list of input files to that bundle, we can create a hash manifest the same way we do for input files in lazyrepo, and then use that to avoid bundling the config file or even loading esbuild next time lazyrepo is invoked.

If the use makes a change to any of the files used to create the bundle it should invalidate the cache and re-bundle the file.

Set up integration test framework

At a minimum we should have a way to create a temp dir on disk and populate it with a pnpm/yarn/npm workspaces setup matching some given config object, then returning a link to the dir to use in tests, along with setting up the workspace root and automating cleanup logic.

Maybe something like

function runTestInProject(config, async (dir) => {
  // do the test logic
})

Look at wireit for possible prior art.

slugify script names

When writing files or doing other encoding of info that needs to be url-safe, we need to slugify script names. This was happening before but got lost at some point.

Also need to deal with slug conflict resolution

Clean up child processes when exiting for any reason

We can probably copy what yarn 2 does here: https://github.com/yarnpkg/berry/blob/cb2eecc859145235dcab6e338b264069c64ba027/packages/yarnpkg-shell/sources/pipe.ts#L28

i.e. keep a reference to any spawned process and make sure they get cleaned up on sigterm and whatever else

Always include lockfile in cache key

The user's lockfile should always be included in every cache key, unless we start doing static analysis of tasks to find out which packages they use like a big boy build system (Let's not do that 😓)

We can add an unsafe option to the CacheConfig type to turn off this behaviour on a per-script basis, maybe unsafe_excludeLockfile

Note that the lockfile path can be configured in yarn. Doesn't look like that's the case with npm/pnpm.

add support for rootDir to baseCommand

The baseCommand option should support the <rootDir> macro.

e.g. I should be able to write

export default {
  tasks: {
    build: {
      baseCommand: 'node <rootDir>/scripts/build.js'
    }
  }
}

and lazyrepo will replace the <rootDir> bit with the absolute path to the root workspace dir when executing this task in any workspace.

Add validation for config file

We should validate configuration file and provide helpful error message if the config doesn't match provided schema.

Use a better logging library

I copied over this log.js file from an old project of mine, but I don't love it. It also doesn't support log levels (e.g. --verbose, --debug, etc).

Let's find something better we can build on. Doesn't matter what it is as long as it's lightweight and makes life easier.

Do an audit of any log statements in the app (console.log is used in a few places too) and replace them with the new logging system.

Allow ignoring files or directories

It'd be quite convenient to ignore some files or directories when running scripts. We can add --ingore option to the CLI root and run commands. This option should be implemented using glob patterns. When choosing scope to run script on we should also take into account value of --filter option.

Examples:

• lazyrepo build --ignore=packages/ui/**/*
• lazyrepo run build --ignore=packages/**/* --ignore=apps/**/*

Add support for TypeScript config file

We should add support for configuration files with ts and mts file extensions. We can use TypeScript API or esbuild to transpile code to JavaScript.

lazy-repo fails when parent of project contains package.json

lazy-repo fail install when yarn installing(cmd yarn) tldraw with the following error message

➤ YN0007: │ @tldraw/monorepo@workspace:. must be built because it never has been before or the last one failed
➤ YN0000: │ @tldraw/monorepo@workspace:. STDOUT husky - Git hooks installed
➤ YN0000: │ @tldraw/monorepo@workspace:. STDOUT lazyrepo 0.0.0-alpha.27
➤ YN0000: │ @tldraw/monorepo@workspace:. STDOUT -----------------------
➤ YN0000: │ @tldraw/monorepo@workspace:. STDOUT ∙ ERROR ∙ Failed reading package.json in '/Users/JohnDoe/tldr'

lazy-repo continues to traverse directories upwards if it finds the parent dir contains package.json

This has been mentioned before in the tldraw project: tldraw/tldraw#1746

If a task fails, re-log it's output at the end, without prefixes

This will involve streaming the output from a task to a file as well as to the terminal, then at the end, if the task failed, spitting the contents of the file onto stdout inside a task group (same as we do for the manifests)

Support custom inputs

We should let folks define async functions that can return an object matching {[key: string]: string | Buffer} that can be fed into manifests for tasks.

There should be a global option and a task-local option.

• globalCustomInputs on LazyConfig
• customInputs on TaskConfig

both with type

(args: {taskName: string, taskDir: string, }) => {[key: string]: string | Buffer} | Promise<{[key: string]: string | Buffer}>

Make stack traces in TS config files clickable

Followup to #4

When lazyrepo loads a .ts config file via esbuild, the filenames in the stacktrackes of any errors thrown are prefixed by file:///private/ for some reason. This makes them unclickable in terminal output.

This task is to find a way to strip those prefixes out. I assume there's a way we can configure esbuild to not do that. If not, it's no big deal.

Add run summary

turbo has this nice run summary that says how many things passed and failed etc, we should have something like that

It can be exactly like this, or completely different. As long as it looks good.

I'd love to keep the silly >>> MAXIMUM LAZY thing but otherwise, have at it!

Log full manifest + diff on CI

We can use grouping for github actions so the output doesn't overwhelm the reader.

https://gist.github.com/Dkendal/065c7713aea782c232bd0f94985504a7

Check whether other CI providers support this or other grouping formats.

We don't need the task key prefix, since they will be logged synchronously, but we should add four spaces to make it easy to skip over if you need to scan the log manually.

We can use the ci-info package to detect whether it's running on CI.

Config file changes should minimally bust the cache

We can hash individual parts of the lazy config file when computing the manifest for a task (in computeManifest.js). Rather than busting the whole cache for every change we can be more targeted and only bust the parts that matter. I think we can get away with just hashing the baseCacheConfig and the current task's script config, but part of this ticket is to think deeply about that.

In addition we should also always include a hash of the "scripts" entry string in the manifest, regardless of where it comes from.

Include `devDependencies`, `peerDependencies`, and `optionalDependencies` in the package dependency graphs

When we collect the local dependencies for a package, we should do it for all of the dependencies fields, not just "dependencies"

lazyrepo/src/workspace.js

Lines 23 to 26 in 1d58bf0

	localDeps: Object.keys(packageJson.dependencies ?? {})
	.filter((dep) => allLocalPackageNames.includes(dep))
	// TODO: This sort is depended upon, test it!!!
	.sort(),

add node version to cache key

We should add the node version to the input manifests.

Add CLI option to control color output

We should add global CLI option that controls color output. It should overwrite any logic that detects TTY context. We can have two options: --color and --no-color. If there is no provided option we can fallback to auto detection of the execution environment.

extraArgs (passed in on the cli) should contribute toward the cache key

If i run lazy test -- foo and it succeeds then it should not necessarily succeed when I run lazy test -- bar.

We should stringify the extraArgs and add them to the manifest in computeManifest.js. We'll need to add another manifest 'type' too

lazyrepo/src/manifest/computeManifest.js

Lines 10 to 15 in 249849f

	const types = {
	upstreamTaskInputs: 'upstream task inputs',
	dependencyTaskInputs: 'dependency task inputs',
	envVar: 'env var',
	file: 'file',
	}

Change throwOnError to expectError

The TestHarness.exec method takes a throwOnError with a default value of true. Let's replace it with an expectError flag with a default value of false, to clarify the intent of the flag.

Replace usage of the name `cwd` in task management code

There are a few places in TaskGraph and elsewhere that use the name cwd to mean 'the dir of the package to execute the task in' and I'd much prefer it to be called taskDir or something.

Fix filter path parsing

Currently lazy test --filter blah is not working, instead you need to do --filter='blah' or similar.

Let's add some regression tests 🙃

fast-glob should always be fed absolute paths

It turns out that globbing (or maybe, rather, micromatch) behaves in odd ways when you mix absolute and relative paths while passing both implicit and explicit cwd. We should make sure we're always giving fast-glob absolute paths, just to be safe. Then the cwd arg becomes moot so it can be elided.

Look into adding a wrapper around glob.sync to assert that all include and ignore patterns are absolute. Audit all existing usages of fast-glob.

Clean way to set defaults for user config options

User config defaults are currently applied ad-hoc in the places where they are used. This is a recipe for disaster.

We should really have two sets of config types: one 'partial' set that users see, and one 'full' set with defaults applied that we use internally. When we load the config file we should covert it from the 'partial' type into the 'full' type, applying the defaults in a single place to avoid inconsistency.

Cache Outputs

Look at how wireit is doing it

What is the difference between lazyrepo, turborepo and rush?

Is it essential to outline the comparison?

Add automatic flake detection

A big problem with turborepo is that when tasks end up running too often, you get no feedback as to why.

Lazyrepo already helps out here by listing what changed from one run to the next.

Another potentially very useful thing to do would be to detect and flag common causes of flakiness.

The biggest cause is when a task modifies one of its own input files, or the input files of one of its upstream dependencies.

It would be fairly minimal overhead to check the mtimes of upstream input files after each task ends. I wonder whether fs.watch would be cheap enough for this purpose too.

Get working on Windows and add CI checks that run on windows

It should work for both WSL and in the main windows shell, whatever that's called. Do people still use that git bash shell?

I haven't done any dev on windows since 2011 so please excuse my ignorance.

Add colors to diff lines

We currently log a truncated list of changes when there's a cache miss that results in a diff file.

lazyrepo/src/runTask.js

Lines 45 to 56 in 1d58bf0

	if (diff?.length) {
	const allLines = diff.split('\n')
	const diffPath = getDiffPath(task)
	if (!existsSync(path.dirname(diffPath))) {
	mkdirSync(path.dirname(diffPath), { recursive: true })
	}
	writeFileSync(diffPath, stripAnsi(allLines.join('\n')))
	print('cache miss, changes since last run:')
	allLines.slice(0, 10).forEach((line) => print(kleur.gray(line)))
	if (allLines.length > 10) {
	print(kleur.gray(`... and ${allLines.length - 10} more. See ${diffPath} for full diff.`))
	}

We should add some subtle color here to provide extra visual cues of what changed to cause the cache miss.

I previously had it so that the +, -, and ± symbols were green, red, and yellow respectively, with the rest of the line remaining gray. That worked for me, but feel free to experiment.

Add auto for publishing + changelogs

Let's use intuit/auto for publishing. copy canary + alpha workflow over from tldraw.

Running `lazy <script>` in a non-root package directory

At the moment this is broken, there's some bad path resolution logic to figure out.

It should work and it should narrow the scope of the script to only run the given task in the given package, and any tasks it depends on obviously.

i.e. if you are in pacakges/core and you run lazy test it should do the same thing as if you are in the root dir and you run lazy :run test packages/core

Add tests! If the integration test framework is not set up yet, at least add a test file with some test.todo('...') entries.

General CLI improvements

I was looking through the source code and was thinking that it might be a good idea to use some lightweight framework for building CLI. It should make it easier to maintain current and add new commands in the future. It would also simplify things like help command, arguments parsing, validation etc.

Please, let me know your thoughts about it.

Include global inputs in manifests

This involves:

• Implementing support for the globalDependencies (let's rename to globalIncludes) and globalExcludes options.
• Adding support for a global env vars option: globalEnvInputs (also rename inputEnvVars in task config to envInputs)
• Thinking about what the default values of globalIncludes and globalExcludes should be, if anything. Not important to get it right immediately, but take a look at what turbo and wireit do at least.

Set up ESLint

Set up ESLint in the repository.

@ds300 Do you have any preferences for rules (apart from the set of recommended rules from eslint) or any config file I can refer to?