Giter Site home page Giter Site logo

0x80 / isolate-package Goto Github PK

View Code? Open in Web Editor NEW
105.0 4.0 11.0 395 KB

Isolate a monorepo package with its internal dependencies to form a self-contained directory with a pruned lockfile

License: MIT License

TypeScript 99.39% JavaScript 0.61%
ci deploy firebase isolate monorepo npm package pnpm workspace yarn

isolate-package's Introduction

Isolate Package

Quickstart

Run npx isolate-package isolate from the monorepo package you would like to isolate.

If you would like to see an example of a modern monorepo with this tool integrated, check out mono-ts

Features

  • Isolate a monorepo workspace package to form a self-contained package that includes internal dependencies and an adapted lockfile for deterministic deployments.
  • Preserve packages file structure, without code bundling
  • Should work with any package manager, and tested with NPM, PNPM, and Yarn (both classic and modern)
  • Zero-config for the vast majority of use-cases
  • Isolates dependencies recursively. If package A depends on internal package B which depends on internal package C, all of them will be included
  • Optionally force output to use NPM with matching versions
  • Optionally include devDependencies in the isolated output
  • Optionally pick or omit scripts from the manifest
  • Compatible with the Firebase tools CLI, including 1st and 2nd generation Firebase Functions. For more information see the Firebase instructions.
  • Available in a forked version of firebase-tools to preserve live code updates when running the emulators

Installation

Run pnpm install isolate-package -D or the equivalent for npm or yarn.

I recommended using pnpm over npm or yarn. Besides being fast and efficient, PNPM has better support for monorepos.

Usage

!! If you plan use this for Firebase deployments, and you want to preserve live code updates when running the local emulators, you will want to use firebase-tools-with-isolate instead.

This package exposes a binary called isolate.

Run npx isolate from the root of the package you want to isolate. Make sure you build the package first.

The isolate binary will try to infer your build output location from a tsconfig file, but see the buildDirName configuration if you are not using Typescript.

By default the isolated output will become available at ./isolate.

If you are here to improve your Firebase deployments check out the Firebase quick start guide.

Troubleshooting

If something is not working as expected, add an isolate.config.json file, and set "logLevel" to "debug". This should give you detailed feedback in the console.

In addition define an environment variable to debug the configuration being used by setting DEBUG_ISOLATE_CONFIG=true before you execute isolate.

When debugging Firebase deployment issues it might be convenient to trigger the isolate process manually with npx isolate and possibly DEBUG_ISOLATE_CONFIG=true npx isolate.

Prerequisites

Because historically many different approaches to monorepos exist, we need to establish some basic rules for the isolate process to work.

Define shared dependencies in the package manifest

This one might sound obvious, but if the package.json from the package you are targeting does not list the other monorepo packages it depends on, in either the dependencies or devDependencies list, then the isolate process will not include them in the output.

How dependencies are listed with regards to versioning is not important, because packages are matched based on their name. For example the following flavors all work (some depending on your package manager):

// package.json
{
  "dependencies": {
    "shared-package": "0.0.0"
    "shared-package": "*",
    "shared-package": "workspace:*",
    "shared-package": "../shared-package",
  }
}

So if the a package name can be found as part of the workspace definition, it will be processed regardless of its version specifier.

Define "version" field in each package manifest

The version field is required for pack to execute, because it is use to generate part of the packed filename. A personal preference is to set it to "0.0.0" to indicate that the version does not have any real meaning.

Define "files" field in each package manifest

NOTE: This step is not required if you use the internal packages strategy but you could set it to ["src"] instead of ["dist"].

The isolate process uses (p)npm pack to extract files from package directories, just like publishing a package would.

For this to work it is required that you define the files property in each package manifest, as it declares what files should be included in the published output.

Typically, the value contains an array with only the name of the build output directory. For example:

// package.json
{
  "files": ["dist"]
}

A few additional files from the root of your package will be included automatically, like the package.json, LICENSE and README files.

Tip If you deploy to Firebase 2nd generation functions, you might want to include some env files in the files list, so they are packaged and deployed together with your build output (as 1st gen functions config is no longer supported).

Use a flat structure inside your packages folders

At the moment, nesting packages inside packages is not supported.

When building the registry of all internal packages, isolate doesn't drill down into the folders. So if you declare your packages to live in packages/* it will only find the packages directly in that folder and not at packages/nested/more-packages.

You can, however, declare multiple workspace packages directories. Personally, I prefer to use ["packages/*", "apps/*", "services/*"]. It is only the structure inside them that should be flat.

Configuration Options

For most users no configuration should be necessary.

You can configure the isolate process by placing a isolate.config.json file in the package that you want to isolate, except when you're deploying to Firebase from the root of the workspace.

For the config file to be picked up, you will have to execute isolate from the same location, as it uses the current working directory.

Below you will find a description of every available option.

logLevel

Type: "info" | "debug" | "warn" | "error", default: "info".

Because the configuration loader depends on this setting, its output is not affected by this setting. If you want to debug the configuration set DEBUG_ISOLATE_CONFIG=true before you run isolate

buildDirName

Type: string | undefined, default: undefined

The name of the build output directory name. When undefined it is automatically detected via tsconfig.json. When you are not using Typescript you can use this setting to specify where the build output files are located.

includeDevDependencies

Type: boolean, default: false

By default devDependencies are ignored and stripped from the isolated output package.json files. If you enable this the devDependencies will be included and isolated just like the production dependencies.

pickFromScripts

Type: string[], default: undefined

Select which scripts to include in the output manifest scripts field. For example if you want your test script included set it to ["test"].

By default, all scripts are omitted.

omitFromScripts

Type: string[], default: undefined

Select which scripts to omit from the output manifest scripts field. For example if you want the build script interferes with your deployment target, but you want to preserve all of the other scripts, set it to ["build"].

By default, all scripts are omitted, and the pickFromScripts configuration overrules this configuration.

omitPackageManager

Type: boolean, default: false

By default the packageManager field from the root manifest is copied to the target manifest. I have found that some platforms (Cloud Run, April 2024) can fail on this for some reason. This option allows you to omit the field from the isolated package manifest.

isolateDirName

Type: string, default: "isolate"

The name of the isolate output directory.

targetPackagePath

Type: string, default: undefined

Only when you decide to place the isolate configuration in the root of the monorepo, you use this setting to point it to the target you want to isolate, e.g. ./packages/my-firebase-package.

If this option is used the workspaceRoot setting will be ignored and assumed to be the current working directory.

tsconfigPath

Type: string, default: "./tsconfig.json"

The path to the tsconfig.json file relative to the package you want to isolate. The tsconfig is only used for reading the compilerOptions.outDir setting. If no tsconfig is found, possibly because you are not using Typescript in your project, the process will fall back to the buildDirName setting.

workspacePackages

Type: string[] | undefined, default: undefined

When workspacePackages is not defined, isolate will try to find the packages in the workspace by looking up the settings in pnpm-workspace.yaml or package.json files depending on the detected package manager.

In case this fails, you can override this process by specifying globs manually. For example "workspacePackages": ["packages/*", "apps/*"]. Paths are relative from the root of the workspace.

workspaceRoot

Type: string, default: "../.."

The relative path to the root of the workspace / monorepo. In a typical setup you will have a packages directory and possibly also an apps and a services directory, all of which contain packages. So any package you would want to isolate is located 2 levels up from the root.

For example

packages
├─ backend
│  └─ package.json
└─ ui
   └─ package.json
apps
├─ admin
│  └─ package.json
└─ web
   └─ package.json
services
└─ api
   └─ package.json

When you use the targetPackagePath option, this setting will be ignored.

API

Alternatively, isolate can be integrated in other programs by importing it as a function. You optionally pass it a some user configuration and possibly a logger to handle any output messages should you need to write them to a different location as the standard node:console.

import { isolate } from "isolate-package";

await isolate({
  config: { logLevel: "debug" },
  logger: customLogger,
});

If no configuration is passed in, the process will try to read isolate.config.json from the current working directory.

The internal packages strategy

An alternative approach to using internal dependencies in a Typescript monorepo is the internal packages strategy, in which the package manifest entries point directly to Typescript source files, to omit intermediate build steps. The approach is compatible with isolate-package and showcased in my example monorepo setup

In summary this is how it works:

  1. The package to be deployed lists its internal dependencies as usual, but the package manifests of those dependencies point directly to the Typescript source (and types).
  2. You configure the bundler of your target package to include the source code for those internal packages in its output bundle. In the case of TSUP for the API service in the mono-ts that configuration is: noExternal: ["@mono/common"]
  3. When isolate runs, it does the same thing as always. It detects the internal packages, copies them to the isolate output folder and adjusts any links.
  4. When deploying to Firebase, the cloud pipeline will treat the package manifest as usual, which installs the listed dependencies and any dependencies listed in the linked internal package manifests.

Steps 3 and 4 are no different from a traditional setup.

Note that the manifests for the internal packages in the output will still point to the Typescript source files, but since the shared code was embedded in the bundle, they will never be referenced via import statements. So the manifest the entry declarations are never used. The reason the packages are included in the isolated output is to instruct package manager to install their dependencies.

Firebase

For detailed information on how to use isolate-package in combination with Firebase see this documentation

isolate-package's People

Contributors

0x80 avatar jieey1140 avatar vajahath avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

isolate-package's Issues

Reintroduce excludeLockfile

In situations where the lockfile isn't working, and fallback to NPM is not an option (PNPM) we still want an escape hatch.

How to use with Turborepo style internal packages where "main" and "types" point to "src/index.ts"?

Have you managed to get this working when packages are not compiled themselves, but instead have main and types pointing directly to src/index.ts?

This style is championed by Turbo Repo on their blog and in the docs for internal packages. The point being we don't need to separately build the package each time. We import it as pure TypeScript and then the app does the transpiling when needed.

I prefer it because I can skip the build step for internal packages, changes appear instantly throughout the monorepo, and it allows for the customization of the tsconfig.json file for each individual app, as opposed to having to having a more generic tsconfig.json version for the packages that aims to cater to all the apps.

But when working with isolate package I cannot get it working because I think it requires that packages are built before they are isolated. Instead I need the internal packages added, and then build everything.

Is that possible?

There are types at 'isolate-package/dist/index.d.ts', but this result could not be resolved when respecting package.json "exports".

In order to solve problem described in #56, I built a script to handle my isolation process. However, TypeScript refuses to import the isolate-package types.

Here is the full error:

Could not find a declaration file for module 'isolate-package'. '[workspace]/node_modules/.pnpm/[email protected]/node_modules/isolate-package/dist/index.mjs' implicitly has an 'any' type.
  There are types at [workspace]/apps/api/node_modules/isolate-package/dist/index.d.ts', but this result could not be resolved when respecting package.json "exports". The 'isolate-package' library may need to update its package.json or typings.ts(7016)

I am using isolate-package v1.10.1.

The script:
import { execSync } from 'node:child_process';
import { readFile, writeFile } from 'node:fs/promises';
import { isolate } from 'isolate-package';
import { rimraf } from 'rimraf';
import { build as tsup } from 'tsup';
import tsupConfig from '../tsup.config';

const isolateDirName = 'isolate';

await rimraf(['dist', isolateDirName]);
await tsup(tsupConfig);
await isolate({
  isolateDirName,
});

// Copy the overrides from the root workspace to the isolate workspace
const rootPackageJson = JSON.parse(await readFile('../../package.json', 'utf-8'));
const isolatePackageJson = JSON.parse(await readFile('package.json', 'utf-8'));
isolatePackageJson.pnpm = { ...isolatePackageJson.pnpm, overrides: rootPackageJson.pnpm.overrides };

await writeFile('isolate/package.json', JSON.stringify(isolatePackageJson, null, 2));
execSync('pnpm install --lockfile-only', { cwd: isolateDirName, stdio: 'inherit' });
execSync('zip -r ../dist/function.zip ./*', { cwd: isolateDirName, stdio: 'inherit' });

await rimraf([isolateDirName]);
My tsconfig.json:
{
  "extends": "@tsconfig/strictest/tsconfig.json",
  "compilerOptions": {
    "outDir": "dist",
    "strict": true,

    "allowJs": false,
    "checkJs": false,
    "noEmit": true,
    "allowImportingTsExtensions": true,

    "forceConsistentCasingInFileNames": true,
    "verbatimModuleSyntax": true,
    "esModuleInterop": true,
    "resolveJsonModule": true,
    "isolatedModules": true,
    "allowSyntheticDefaultImports": true,

    "exactOptionalPropertyTypes": false,

    "target": "ES2022",
    "module": "ESNext",
    "moduleResolution": "Bundler",

    "paths": {
      "@/*": ["./src/*"]
    }
  },
  "include": ["**/*.ts"],
  "exclude": ["node_modules"]
}

The fix is fairly easy: add explicit types to the package.json exports:

   "type": "module",
   "types": "./dist/index.d.ts",
   "exports": {
-    ".": "./dist/index.mjs"
+    ".": {
+      "import": "./dist/index.mjs",
+      "types": "./dist/index.d.ts"
+    }
   },
   "files": [
     "dist",

Happy to submit a PR.

My solution to the NPM package-lock.json problem

I encountered deployment issues with npm due to the package-lock.json file.

Console messages indicated missing packages within the package-lock.json. Though excluding this file from the configuration allowed for smooth deployment, I wanted to retain it.

To identify the problematic package, I began removing packages individually. For me, the issue stemmed from firebase-functions-test—an old package I no longer utilized. After uninstalling it, the deployment went smoothly.

If you're facing a similar issue, I recommend removing packages individually to identify the problematic one.

I hope future versions of isolate-package become universally compatible. Until then, this method might be a workaround.

Lastly, if you pinpoint a problematic package, please share it here to aid others in faster troubleshooting!

use within npm workspace

I'm having trouble getting this working within a project that uses npm workspace. I have a workspace defined in a package.json file at the root:

  ...
  "workspaces": [
    "packages/*",
    "firebase/*"
  ],
  ...

And in the firebase directory I have a functions subdirectory (containing my firebase.json file). And inside packages there is a directory containing a typescript library called shared-code which is declared as a dependency in the package.json file in the functions.

root
  -- tsconfig.json
     package.json (defines workspaces "firebase" and "packages")
     node_modules

  -- firebase
    -- functions
      -- firebase.json
         package.json (with dependency file:../../shared-code)
         <-- no node_modules! -->

  -- packages
    -- shared-code
      -- package.json
         tsconfig.json
         node_modules

When I run npm install in the root of the entire project, the result is that a node_modules directory gets created at the root of the project, BUT there is no node_modules directory at all inside my functions workspace.

So when I run firebase deploy ... (inside firebase/functions) it fails immediately because there is no node_modules dir present. In my case with Cannot find local ESLint!.

@0x80 Is there a solution?

Isolate seem to be not copying the `.env` file

Isolate seem to be not copying the .env file. (but it copies .env.local and .env.staging in my case)

According to firebase docs,

If you need an alternative set of environment variables for your Firebase projects (such as staging vs production), create a .env.<project or alias> file and write your project-specific environment variables there. The environment variables from .env and project-specific .env files (if they exist) will be included in all deployed functions.

(Also see the tabled example from the docs to see how the envs are inherited)

In short, .env is a valid env config for firebase functions.

Shared package deep dependencies build wrong file:PATHs.

Hi!

I'm experiencing an issue.

I have a dependency graph that looks like this:

functions <- utils package.
utils package <- other_shared_package.

Running isolate succesfully puts both packages side by side in a folder structure like:

./isolate
--./dist
--./packages
----./utils
----./other_shared_package

Inspecting the package.json of these packages in the isolate folder I see that the utils package has this dep (isolate/packages/utils/package.json):

"dependencies": {
   "other_shared_package": "file:packages/other_shared_package",
}

This causes an error when running yarn install:
"Package "other_shared_package" refers to a non-existing file '"[long_monorepo_path]/isolate/packages/utils/packages/other_shared_package"'."

So it looks like yarn thinks that the file should be relative to the utils folder. Suspecting this, I tried to modify the dependency to be

"dependencies": {
    "other_shared_package": "file:../other_shared_package",
}

Which resulted in a succesful install!

So it seems like the isolate process is not compatible with yarn (I'm running yarn v.1.22.17)?

I'm also experiencing the same issue when trying to deploy to the firestore function service (they try to install using yarn v1.22.19), but debugging there takes a lot longer than debugging locally.

Thanks again for the package. I'm going to see if I can maybe force firestore to install with npm and hopefully that can serve as a temp fix!

Firebase functions reloading not working

Hey. So perhaps this is a misconfiguration on my part, but firebase code reloading is not working for me when isolate-package is involved.

Excerpt from firebase.json:

{
  "functions": [
    {
      "source": "./isolate",
      "runtime": "nodejs20",
      "codebase": "default",
      "ignore": [
        "node_modules",
        ".git",
        "firebase-debug.log",
        "firebase-debug.*.log"
      ],
      "predeploy": [
        "npm run build",
        "npm run isolate"
      ]
    }
  ]
}

So, npm run build causes tsc to emit into ./dist and then npm run isolate bundles that all up into ./isolate. So far so good, the emulators boot up fine and my functions handle requests as expected.

If I now make some changes and run npm run build, obviously nothing will change in the emulators as they are not watching that directory. Then, if I then run npm run isolate again, I would expect the emulators to see the changes and cycle in the new code - but it does not. At this point I have to manually kill and restart the emulators to see the new code changes take effect.

I believe the cause of this behaviour is firebase-tools using chokidar to watch the ./isolate/dist directory. Every time the isolate command is run, it removes the ./isolate directory entirely before replacing it with the new isolated bundle. The removal of the ./isolate/dist directory seems to cause chokidar to stop tracking changes, even though a new ./isolate/dist directory is created shortly after.

There are a couple of unanswered issues in the chokidar issue tracker that talk about this behaviour, so I don't expect it be resolved there or in firebase-tools any time soon. And since this seems to be an issue because of the way the isolate command works, I suspect it would be easiest to fix here - probably by not deleting isolate or isolate/dist. The dist part is specific to my project too I suspect (derived from main in package.json?), so that would need generalising too.

I suppose one other workaround would be to remove isolate from firebase.json entirely, and only use it when deploying but that feels like more of a hack and would mean I was testing with code that wasn't necessarily identical to that which would be in the isolated bundle.

Add support for pnpm v9

After upgrading to pnpm v9, the isolate script is now failing. Here's what I see in the logs:

i  functions: Start isolating the source folder...
info Generating PNPM lockfile...
i  functions: +++ Failed to isolate: Failed to generate lockfile: value.startsWith is not a function

I know that the lockfile format has changed in PNPM v9. Could it be related to that?

Error on including monorepo dependencies with import statements

First of all, thank you for your work on this awesome project.

I'm trying to use my local package which is a Prisma (DB) client so I can use it in my firebase function (I'm starting based on this T3-Turbo template).

When npx isolate is run as part of the deploy, it copies over my package exactly as it is in my typescript project (it copies index.ts with import statements). However I believe I need my shared package to be transpiled to commonJS for it to run in the firebase cloud environment and it doesn't seem like there's a way to do this.

The error I get is SyntaxError: Cannot use import statement outside a module which seems to from db/index.ts where I have the line import { PrismaClient } from "@prisma/client"

Do you have any idea why this would happen?

Here's the last stage of my deploy where it fails:

debug Linking dependency @acme/db to file:packages/db
debug Copied lockfile to /Users/jjdevereux/code/reasily-monorepo/packages/functions/isolate/package-lock.json
debug Deleting temporary directory (root)/packages/functions/isolate/__tmp
debug Stored isolate output at /Users/jjdevereux/code/reasily-monorepo/packages/functions/isolate
info Isolate completed
✔  functions: Finished running predeploy script.
i  functions: preparing codebase default for deployment
i  functions: ensuring required API cloudfunctions.googleapis.com is enabled...
i  functions: ensuring required API cloudbuild.googleapis.com is enabled...
i  artifactregistry: ensuring required API artifactregistry.googleapis.com is enabled...
✔  functions: required API cloudbuild.googleapis.com is enabled
✔  artifactregistry: required API artifactregistry.googleapis.com is enabled
✔  functions: required API cloudfunctions.googleapis.com is enabled

Error: Failed to load function definition from source: Failed to generate manifest from function source: SyntaxError: Cannot use import statement outside a module
npm ERR! Lifecycle script `deploy` failed with error: 
npm ERR! Error: command failed 
npm ERR!   in workspace: [email protected] 
npm ERR!   at location: /Users/jjdevereux/code/reasily-monorepo/packages/functions 

Package names with periods don't work (e.g. "@mylib/a.b")

In create-packages-registry.ts, the registry is constructed which indexes package manifests by their name. However, the registry object is built using lodash.set, which parses the key as a recursive path when it includes periods, so that package name "@mylib/a.b" results in { "@mylib/a": { "b": { ... }}}. Later, when "@mylib/a.b" is looked up in the registry, it's not found resulting in the error error TypeError: Cannot read properties of undefined (reading 'rootRelativeDir')

It appears that the extra behavior of lodash.set is not needed. If so, the following change fixes this issue:

.reduce<PackagesRegistry>(
    (acc, info) => {
      if (info) {
        acc[info.manifest.name] = info;
      }
      return acc;
    },
    {}
  );

Support for default folder structure generated by `firebase init`

First off, thank you for this cool tool. ⭐
I thought I could give some feedback here. (Also I thought like this should be a part of turborepo!)

Support for default firebase init generated folder structure.

The firebase init command generates a folder structure similar to:

firebase
├── functions/
└── firebase.json

The functions source will go into functions dir.

The most simplest approach I found for adopting this into a monorepo (I'm using pnpm+turbo) is copy this firebase directory into mono/apps.

mono
├── apps
│      ├── web
│      ├── firebase  <-----------------(the firebase directory copied)
│      └── my-another-app
├── packages/
└── pnpm-workspace.yml

And my pnpm-workspace.yml looked like:

packages:
  - "packages/*"
  - "apps/*"
  - "apps/firebase/functions"
  - "!**/test/**"

Unfortunately this setup seem to be not compatible with isolate. As in the docs of isolate, nested packages are not supported. But the firebase directory is not a package (it doesn't have a package.json, but firebase/functions is). I don't know if we call this a nested package, but isolate couldn't out-of-the-box support firebase init folder structure in a monorepo.

So tried to flat it out, copied the content for mono/firebase/functions into mono/firebase including the package.json. And it worked and I did a real deployment ⚡ .

But. I had to fiddle around the .env management & firebase emulator setup. Previously during development, firebase functions emulator reloads the changed functions automatically after the tsc build. Now I lost that, even after adding the isolate after the tsc. So now every time I have to stop the emulator, build and tsc ,start again to reflect the changes locally. This was a little painful.

So it would be really cool, if isolate support the native firebase folder structure so that devs feel home at the firebase directory.

Github Actions Failing

Thanks for this great package @0x80!

Unfortunately I keep having a critical error when deploying using Github actions.
Locally isolate-package is working like a charm. However, using Github CI crashed because it cannot file a packages (.tgz) file.

@packages/server:deploy: Error: ENOENT: no such file or directory, open 'api-4.1.1.tgz' @packages/server:deploy:  ELIFECYCLE  Command failed with exit code 1. @packages/server:deploy: ERROR: command finished with error: command (/my/private/path) pnpm run deploy exited (1)

I am kinda stuck here. Would you know what is going on?

//isolate.config.json { "buildDirName": "./", "excludeLockfile": true }

Failed to load function definition from source

Any clue on why the following is occuring?

Failed to load function definition from source: FirebaseError: Failed to find location of Firebase Functions SDK. Please file a bug on Github

This is when I try to run with emulators.

My setup:

firebase.json

...
"functions": {
  "source": "./apps/functions/isolate",
  "predeploy": ["turbo build", "pnpm isolate"],
  "runtime": "nodejs18"
},
...

Note: I have my firebase config files at the root of my monorepo

isolate.config.json

{
  "targetPackagePath": "./apps/functions"
}

apps/functions/package.json

...
"engines": {
  "node": "18"
},
"main": "./dist/index.js",
"files": [
  "dist"
],
"scripts": {
  "build": "tsc"
},
devDependencies: {
  ...
  "isolate-package": "^1.3.3"
}
...

apps/functions/src/index.ts

import { initializeApp } from 'firebase-admin/app'
import { onWordCreated } from './words'

initializeApp({
  projectId:
    process.env.GCLOUD_PROJECT === 'demo-words' ? 'demo-words' : undefined
})

export { onWordCreated }

When I try firebase deploy --only functions, I also get:

Running command: npx isolate npm ERR! could not determine executable to run

Option to prevent move of node_modules for NPM

I use Windows, and I often have processes running that have open files in node_modules which prevent it from being moved. Rather than always killing those processes, I did some experimentation with a fork of isolate-package. Interestingly, skipping the move altogether seems to work fine. Arborist found the dependent packages, even though they was several levels above the isolate folder.

Could we add a configuration option which disables the move?

Thanks!

Fails to handle spaces in file paths

I'm working with a standard turbo/yarn workspaces setup as can be found in https://github.com/vercel/turbo/tree/main/examples/with-tailwind and want to use isolate-package to isolate apps/functions (firebase functions setup as outlined in the docs here).

Even after I gave it several tries and went over my whole setup to make sure I'm in line with both the standard turbo/yarn workspaces as well as isolate-package setup, I do not manage though to understand what's going wrong when running isolate, so let me share some code here.

First the actual error which occurs when I run isolate. The problem occurs with the common package (the only package that is needed from ./packages/*) and it seems as the path includes some parts twice with a weird cut or missing piece where usually is a whitespace (in Own Projects):

⋊> ~/L/D/O/l/a/functions on main ⨯ isolate                                                                                                                                     20:02:31
debug Running isolate-package version 1.3.3
debug Found tsconfig at: ./tsconfig.json
debug Workspace root resolved to /Users/<username>/Own Projects/<project>
debug Isolate target package (root)/apps/functions
debug Isolate output directory (root)/apps/functions/isolate
debug Cleaned the existing isolate output directory
debug Detected package manager yarn 3.6.4
debug Override workspace packages via config: packages/*,apps/*
debug Registering package ./packages/ui-interactive
debug Registering package ./packages/ui
debug Registering package ./packages/tsconfig
debug Registering package ./packages/tailwind-config
debug Registering package ./packages/eslint-config-custom
debug Registering package ./packages/common
debug Registering package ./apps/web
debug Registering package ./apps/functions
error Error: Command failed: npm pack --pack-destination /Users/<username>/Own Projects/<project>/apps/functions/isolate/__tmp
npm WARN Ignoring workspaces for specified package(s) 
npm ERR! code ENOENT
npm ERR! syscall open
npm ERR! path /Users/<username>/Own Projects/<project>/packages/common/Projects/<project>/apps/functions/isolate/__tmp/package.json
npm ERR! errno -2
npm ERR! enoent ENOENT: no such file or directory, open '/Users/<username>/Own Projects/<project>/packages/common/Projects/<project>/apps/functions/isolate/__tmp/package.json'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent 

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/<username>/.npm/_logs/2023-10-21T18_02_37_406Z-debug-0.log

    at ChildProcess.exithandler (node:child_process:419:12)
    at ChildProcess.emit (node:events:513:28)
    at maybeClose (node:internal/child_process:1091:16)
    at Socket.<anonymous> (node:internal/child_process:449:11)
    at Socket.emit (node:events:513:28)
    at Pipe.<anonymous> (node:net:322:12)

... the invalid path is the following: /Users/<username>/Own Projects/<project>/packages/common/Projects/<project>/apps/functions/isolate/__tmp/package.json which is a weird combination of /Users/<username>/Own Projects/<project>/packages/common/ and Projects/<project>/apps/functions/isolate/__tmp/package.json where the latter path lacks the beginning and the Own in Own Projects

Now let me share my setup:

./package.json:

{
  "private": true,
  "workspaces": [
    "apps/*",
    "packages/*"
  ],
  "scripts": {
    "build": "turbo build",
  },
  "prettier": "@vercel/style-guide/prettier",
  "devDependencies": {
    "@types/node": "^20.8.7",
    "@types/react": "^18.2.29",
    "@types/react-dom": "^18.2.14",
    "prettier": "^3.0.3",
    "prettier-plugin-tailwindcss": "^0.5.3",
    "tsconfig": "*",
    "turbo": "latest",
    "typescript": "^5.2.2"
  },
  "packageManager": "[email protected]"
}

apps/functions/package.json:

{
  "name": "functions",
  "version": "0.0.0",
  "scripts": {
    "build": "tsc -b tsconfig.build.json",
  },
  "engines": {
    "node": "18"
  },
  "main": "lib/index.js",
  "dependencies": {
    "common": "*",
    "firebase-admin": "^11.8.0",
    "firebase-functions": "^4.3.1"
  },
  "devDependencies": {
    "firebase-tools": "latest",
    "isolate-package": "^1.3.3",
    "typescript": "^4.9.0"
  },
  "private": true
}

apps/functions/isolate.config.json:

{
  "workspaceRoot": "../..",
  "workspacePackages": ["packages/*", "apps/*"],
  "logLevel": "debug"
}

apps/functions/tsconfig.json:

{
  "compilerOptions": {
    "module": "commonjs",
    "noImplicitReturns": true,
    "noUnusedLocals": true,
    "outDir": "lib",
    "sourceMap": true,
    "strict": true,
    "target": "es2017"
  },
  "compileOnSave": true,
  "include": [
    "src",
    "./*.d.ts"
  ]
}

apps/functions/tsconfig.build.json:

{
  "extends": "./tsconfig.json",
  "compilerOptions": {
    "rootDir": "src"
  },
}

packages/common/package.json:

{
  "name": "common",
  "version": "0.0.0",
  "private": true,
  "license": "MIT",
  "exports": {
    ".": "./dist"
  },
  "types": "./dist/index.d.ts",
  "files": [
    "dist"
  ],
  "scripts": {
    "build": "tsup",
    "dev": "tsup --watch",
    "lint": "eslint src/",
    "typecheck": "tsc --noEmit"
  },
  "dependencies": {
    "react": "^18.2.0",
    "react-dom": "^18.2.0"
  },
  "devDependencies": {
    "react": "^18.2.0",
    "tsconfig": "*",
    "tsup": "^6.1.3",
    "typescript": "^5.1.6"
  }
}

packages/common/tsconfig.json:

{
  "extends": "tsconfig/react-library.json",
  "include": ["."],
  "exclude": ["dist", "build", "node_modules"]
}

packages/common/tsup.config.ts:

import { defineConfig, Options } from 'tsup';

export default defineConfig((options: Options) => ({
  treeshake: true,
  splitting: true,
  entry: ['src/**/*.ts'],
  outDir: 'dist',
  format: ['esm'],
  dts: true,
  minify: true,
  clean: true,
  external: ['react'],
  ...options,
}));

`yarn install` in the isolated package to fix lockfile

Currently, yarn lockfiles are copied to the isolated package as-is. However, I think there's an easy solution. Running yarn install in the isolated package will prune unused entries from the lockfile. I believe the resulting lockfile will pin the dependencies of the isolated package to the same dependencies that were pinned in the monorepo. It will also pin the file: dependencies as expected, so there shouldn't be any issue installing the package with --frozen-lockfile in a deploy environment.

It's important that yarn doesn't consider the isolated package as a workspace, but otherwise it seems sound.

Does this seem sensible?

I'll test this in my own repo by running yarn install in a script after isolate. My use case is also Firebase functions.

Unable to deploy with lockfile using PNPM

Build failed: installing pnpm dependencies: %!w(*buildererror.Error=&{  2 2 8622936f ERR_PNPM_OUTDATED_LOCKFILE  Cannot install with "frozen-lockfile" because pnpm-lock.yaml is not up to date with package.json

Note that in CI environments this setting is true by default. If you still need to run install in such cases, use "pnpm install --no-frozen-lockfile"

As a temporary workaround there is configuration option excludeLockfile, which will exclude the lockfile from the isolate output, but as a result the deployment would be possibly non-deterministic.

Patched dependencies are breaking GCP deployments using pnpm

I'm using isolate-package in a pnpm monorepo.

When I'm isolating the functions, the isolated pnpm-lock.yaml includes all patchedDependencies from the root pnpm-lock.

When deploying the functions to GCP gen2 cloud functions, the builder is installing with 'pnpm install --frozen-lockfile' which results in an error because of the patchedDependencies.

It would be great if there was a built in way to exclude the patchedDependencies (in my case, this patch is from my frontend code which has no business breaking my backend). Right now I had to run a custom node script to remove the patched deps after isolate.

Where pnpm is the package manager for the repo, would it makes sense to use 'pnpm pack' instead of 'npm pack'?

We are trying to train people to use pnpm, and one of my devops guys inserted a script that intercepts npm (even though it's installed). That results in this:

(venv) gitpod /workspace/costvine/jspackages/server (dev) $ isolate
error The response from pack could not be resolved to an existing file: /workspace/costvine/jspackages/server/isolate/__tmp/Use pnpm instead of npm :)
Error: ENOENT: no such file or directory, open '/workspace/costvine/jspackages/server/isolate/__tmp/Use pnpm instead of npm :)'

I know npm pack will probably work just as well here, but I'm thinking maybe it would be cleaner to use pnpm pack in this case.

This is really just a prompt for possible discussion. I'm going to try to work around it. I ran into similar problems with the Firebase emulator, but it still worked in our pnpm monorepo, with only a bundler, despite tripping over the fake npm command. I've left the stub for npm in there partly just to see what's calling it, which is more things than I expected.

But I don't want Firebase deploy using npm to install my dependencies, and I need that lockfile to make absolutely sure it builds what we tested. If I could just generate that pruned lockfile, I could probably do the rest with esbuild. There's an old project in the pnpm repository to generate stand-alone lock files, but it's deprecated, and didn't work for me, for different reasons. The instructions say to use pnpm deploy instead, but I can't figure out any way to get that to generate a lock file.

Missing `@pnpm/logger` dependency (peer dependency of `@pnpm/lockfile-file`)

I'm running isolate in a yarn (v1) workspace monorepo, and getting Error: Cannot find module '@pnpm/logger'. It seems this is a peer dependency of @pnpm/lockfile-file, which is a dependency of isolate-package.

It's not listed as a dependency in this repo, so it never ends up being installed in mine.

It seems like a simple fix to add @pnpm/logger@^5.0.0 as a dependency in this repo.

If workspaces is set to an object, we get a crash.

I have this object as my workspaces setting in my root package.json

"workspaces": { "packages": [ "apps/*", "packages/*" ], "nohoist": [ "**/ts-node" ] },

This causes an error on this line: https://github.com/0x80/isolate-package/blob/main/src/helpers/create-packages-registry.ts#L61, as flatMap is not defined on an object.

I have fixed my issue by providing it trough setting workspacePackages in the config manually. Thanks a lot for the project!

You rock!

Dot Env Variables

I'm not sure if this is an issue with this library or me not using it correctly, but my .env variables are not getting deployed along with everything else.

Other than that, everything is working great and there are no error messages or anything. Just silently not picking up the contents of .env.prod.

Not sure how to use the local package

In my expressjs server that I'm using with isolate to deploy on GAE I do imports like

import app from "@/app"

with in the tsconfig:

"@/*": "./src/*"

but after compiling, and running isolate, I have this bug when deploying:

2023-12-01 13:17:04 default[20231201t131511]  Error: Cannot find module '@/app'
2023-12-01 13:17:04 default[20231201t131511]  Require stack:
2023-12-01 13:17:04 default[20231201t131511]  - /workspace/dist/server.js
2023-12-01 13:17:04 default[20231201t131511]      at Module._resolveFilename (node:internal/modules/cjs/loader:1077:15)
2023-12-01 13:17:04 default[20231201t131511]      at Module._load (node:internal/modules/cjs/loader:922:27)

is there a way for this to work?

here is my tsconfig.json (the lib-gouach-db-schemas is my local monorepo lib that I'm including in the project)

{
  "references": [{ "path": "../lib-gouach-db-schemas/tsconfig.cjs.json" }],
  "compilerOptions": {
    "target": "ES2020",
    "module": "CommonJS",
    "moduleResolution": "Node",
    "esModuleInterop": true,
    "allowSyntheticDefaultImports": true,
    "strict": true,
    "sourceMap": true,
    "allowJs": true,
    "skipLibCheck": true,
    "lib": ["esnext.asynciterable"],
    "outDir": "dist",
    "rootDir": "src",
    "baseUrl": ".",
    "paths": {
      "@/*": ["./src/*"],
      "@gouach/db-schemas/*": ["../lib-gouach-db-schemas/dist/cjs/*"]
    },
    "pretty": true,
    "composite": false
  },
  "compileOnSave": true,
  "include": ["src", "src/**/*.js", "src/**/*.ts"]
}

No binaries found in isolate / Failed to detect package manager

Ran pnpx isolate and getting the following error:

.../Library/pnpm/store/v3/tmp/dlx-4070 | +1 +
 ERR_PNPM_DLX_NO_BIN  No binaries found in isolate
.../Library/pnpm/store/v3/tmp/dlx-4070 | Progress: resolved 1, reused 0, downloaded 0, added 0

Running pnpm isolate or npx isolate

error Error: Failed to detect package manager
at detectPackageManager (file:///ROOT/node_modules/.pnpm/[email protected]/node_modules/isolate-package/src/helpers/detect-package-manager.ts:19:9)
at start (file:///ROOT/node_modules/.pnpm/[email protected]/node_modules/isolate-package/src/index.ts:59:26)

Folder structure:

apps
└─ native
   ├─ package.json
   └─ .eslintrc.js
packages
└─ eslint-config-custom
   ├─ index.js
   └─ package.json
functions
├─ lib/index.js
├─ firebase.json
├─ tsconfig.json
└─ package.json

Getting a ERR_MODULE_NOT_FOUND during firebase deploy

Using your demo mono-ts repo as an example, I have attempted to integrate my own library package and deploy firebase functions. On deploy, it makes the isolate folder but as soon as firebase analyses the folder it gives me a ERR_MODULE_NOT_FOUND for one of the external packages in the library package.

info Isolate completed at /mono-ts/services/fns/isolate
✔  functions: Finished running predeploy script.
i  functions: preparing codebase fns for deployment
i  functions: ensuring required API cloudfunctions.googleapis.com is enabled...
i  functions: ensuring required API cloudbuild.googleapis.com is enabled...
i  artifactregistry: ensuring required API artifactregistry.googleapis.com is enabled...
✔  functions: required API cloudfunctions.googleapis.com is enabled
✔  functions: required API cloudbuild.googleapis.com is enabled
✔  artifactregistry: required API artifactregistry.googleapis.com is enabled
i  functions: Loading and analyzing source code for codebase fns to determine what to deploy
Serving at port 8449

Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'ssh2-sftp-client' imported from /mono-ts/services/isolate/dist/index.js
Did you mean to import [email protected]/node_modules/ssh2-sftp-client/src/index.js?
    at new NodeError (node:internal/errors:406:5)
    at packageResolve (node:internal/modules/esm/resolve:789:9)
    at moduleResolve (node:internal/modules/esm/resolve:838:20)
    at defaultResolve (node:internal/modules/esm/resolve:1043:11)
    at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:383:12)
    at ModuleLoader.resolve (node:internal/modules/esm/loader:352:25)
    at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:228:38)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:85:39)
    at link (node:internal/modules/esm/module_job:84:36) {
  code: 'ERR_MODULE_NOT_FOUND'
}

Lock file reported as out of sync in firebase build

Hi, i'm trying to get away from manually copying local dependencies in a firebase functions monorepo and came across this tool which seems to abstract away the work really nicely, thanks for your efforts 👍.

My issue is that copying the lock file in isolate makes the firebase build fail with the following message.

npm ERR! `npm ci` can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.

I tried doing a clean install with no luck. Is this something you can help with? For now, i have it working with the excludeLockfile flag on, but would like to steer away as the deps tree grows.

My environment (created lock file here, and matched it in gcp build)

❯ node -v
v16.19.1
❯ npm -v
8.19.4

Yarn v3: The lockfile would have been modified by this install, which is explicitly forbidden.

Thanks for making this package because it's been sorely needed for a long time for FIrebase Functions deployment. I have most of it working based on your instructions, but yarn install --immutable is breaking on deployments.

Error

In the below example, @arc/feature-flag is a workspace that was relocated to functions/isolate/modules/feature-flag from modules/feature-flag by yarn isolate. The error shows up when we attempt to deploy to firebase functions.

Logs from Google Cloud Build triggered by Firebase deploy:

INFO 2023-08-21T22:16:21.352014524Z Step #2 - "build": Installing Yarn v3.6.1
INFO 2023-08-21T22:16:21.352389680Z Step #2 - "build": 2023/08/21 22:16:21 [DEBUG] GET https://repo.yarnpkg.com/3.6.1/packages/yarnpkg-cli/bin/yarn.js
INFO 2023-08-21T22:16:21.616034608Z Step #2 - "build": DEBUG: Setting environment variable PATH=/layers/google.nodejs.yarn/yarn_engine/bin:/layers/google.nodejs.runtime/node/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
INFO 2023-08-21T22:16:21.660331912Z Step #2 - "build": WARNING: Skipping adding auth token to /www-data-home/.yarnrc.yml. Unable to find Artifact Registry in /workspace/.yarnrc.yml.
INFO 2023-08-21T22:16:21.660349522Z Step #2 - "build": --------------------------------------------------------------------------------
INFO 2023-08-21T22:16:21.660351700Z Step #2 - "build": Running "yarn install --immutable"
INFO 2023-08-21T22:16:22.540249844Z Step #2 - "build": ➤ YN0000: ┌ Resolution step
INFO 2023-08-21T22:16:23.150742724Z Step #2 - "build": ➤ YN0013: │ @arc/feature-flag@file:./modules/feature-flag#./modules/feature-flag::hash=7122b8&locator=%40arc%2Ffunctions%40workspace%3A. can't be found in the cache and will be fetched from the disk
INFO 2023-08-21T22:16:23.310145669Z Step #2 - "build": ➤ YN0000: └ Completed in 0s 769ms
INFO 2023-08-21T22:16:23.518625341Z Step #2 - "build":
INFO 2023-08-21T22:16:23.519048711Z Step #2 - "build": ➤ YN0000: ┌ Post-resolution validation
INFO 2023-08-21T22:16:23.519058767Z Step #2 - "build": ➤ YN0000: │ @@ -217,10 +217,9 @@
INFO 2023-08-21T22:16:23.519060244Z Step #2 - "build": ➤ YN0000: │ linkType: hard
INFO 2023-08-21T22:16:23.519062243Z Step #2 - "build": ➤ YN0000: │
INFO 2023-08-21T22:16:23.519063426Z Step #2 - "build": ➤ YN0000: │ "@arc/feature-flag@file:./modules/feature-flag::locator=%40arc%2Ffunctions%40workspace%3A.":
INFO 2023-08-21T22:16:23.519072723Z Step #2 - "build": ➤ YN0000: │ version: 1.0.0
INFO 2023-08-21T22:16:23.519751074Z Step #2 - "build": ➤ YN0028: │ - resolution: "@arc/feature-flag@file:./modules/feature-flag#./modules/feature-flag::hash=7b4bc8&locator=%40arc%2Ffunctions%40workspace%3A."
INFO 2023-08-21T22:16:23.519999320Z Step #2 - "build": ➤ YN0028: │ - checksum: 9697a40672390184a2b0cc3d42f437a1e56e9e63b3fc24658793cdacec3681b95a373d13bc856b90b1cb9c2ca7cb337b4d2916c9cdaae5ddf6ae808e8fdec42f
INFO 2023-08-21T22:16:23.520005499Z Step #2 - "build": ➤ YN0028: │ + resolution: "@arc/feature-flag@file:./modules/feature-flag#./modules/feature-flag::hash=7122b8&locator=%40arc%2Ffunctions%40workspace%3A."
INFO 2023-08-21T22:16:23.520006863Z Step #2 - "build": ➤ YN0000: │ languageName: node
INFO 2023-08-21T22:16:23.520298259Z Step #2 - "build": ➤ YN0000: │ linkType: hard
INFO 2023-08-21T22:16:23.520474017Z Step #2 - "build": ➤ YN0000: │
INFO 2023-08-21T22:16:23.520479605Z Step #2 - "build": ➤ YN0000: │ "@arc/functions@workspace:.":
INFO 2023-08-21T22:16:23.520945843Z Step #2 - "build": ➤ YN0000: │
INFO 2023-08-21T22:16:23.521661955Z Step #2 - "build": ➤ YN0028: │ The lockfile would have been modified by this install, which is explicitly forbidden.
INFO 2023-08-21T22:16:23.521782341Z Step #2 - "build": ➤ YN0000: └ Completed in 0s 205ms
INFO 2023-08-21T22:16:23.522244724Z Step #2 - "build": ➤ YN0000: Failed with errors in 0s 987ms

Setup

  • Yarn v3.6.1 with multiple workspaces.
  • firebase.json configured explicitly to ignore .yarn/cache and node_modules because including them would violate the 100MB Firebase Function zip file limit.

isolate.config.json:

{
  "includeDevDependencies": true,
  "targetPackagePath": "./functions",
  "workspaceRoot": "./"
}

firebase.json:

{
  "functions": {
    "source": "functions/isolate",
    "ignore": [
      "**/.yarn/cache/**",
      "**/.yarn/install-state.gz",
      "**/.git",
      "**/.runtimeconfig.json",
      "**/firebase-debug.log",
      "**/firebase-debug.*.log",
      "**/node_modules/**"
    ]
  }
}

Pre-deploy steps in CI job

yarn isolate
cp -r .yarn ./functions/isolate/.yarn
cp .yarnrc.yml ./functions/isolate/.yarnrc.yml
cd ./functions/isolate

# On hindsight this probably does not do anything because Cloud Build explicitly 
yarn config set enableImmutableInstalls falseruns yarn install --immutable
yarn install --mode=update-lockfile 

yarn firebase deploy -P $FIREBASE_PROJECT --only functions --force --debug

Question: Still unclear how to setup my firebase in turborepo.

Hi! First of all, thank you for putting this together! It looks like a very useful tool, especially when it comes to using Firebase with monorepos.

I'll be honest, this is the first time I'm working with Firebase, so the experience there isn't much, but I do have a good understanding of Turborepo monorepos.

I've read both your blog post and the readme in this repo and, even though I do understand the problem, I'm having a hard time using this to set up Firebase for my needs.

Imagine the following example monorepo:

turbo.json
package.json
apps/
    nextjs/
        package.json
        src/
    another/
        package.json
        src/
packages/
    api/
        package.json
        src/
   ui/
       package.json
       src/
tooling/
    firebase/
      firebase.json
      package.json

Assuming the scenario described above, I'd like to be able to:

  • deploy the apps/nextjs app to Firebase (hosting + functions)
  • deploy the packages/api package to Firebase (this package contains our backend API in form of cloud functions)
  • tooling/firebase simply contains emulator configuration and possibly other firebase config files to be used across the repo (basically Firebase config that is not deployed, per se)

Is this scenario a possibility? How would I go about configuring this? I'm assuming this means multiple "target packages"?

Thanks in advance for the help!

P.S.: I'm using pnpm, if that matters

Could not install from "../../apps/functions/isolate/__tmp" as it does not contain a package.json file

Can't isolate packages

Screen Shot 2023-06-08 at 1 17 12 PM

NPM log

1 verbose cli [
1 verbose cli   '/Users/mac/.nvm/versions/node/v14.21.3/bin/node',
1 verbose cli   '/Users/mac/.nvm/versions/node/v14.21.3/bin/npm',
1 verbose cli   'pack',
1 verbose cli   '--pack-destination',
1 verbose cli   '/Users/mac/Projects/suscripciones/apps/functions/isolate/__tmp'
1 verbose cli ]
2 info using [email protected]
3 info using [email protected]
4 verbose npm-session 3bb98f59d50c53a9
5 silly fetchPackageMetaData error for /Users/mac/Projects/suscripciones/apps/functions/isolate/__tmp Could not install from "../../apps/functions/isolate/__tmp" as it does not contain a package.json file.
6 verbose stack Error: ENOENT: no such file or directory, open '/Users/mac/Projects/suscripciones/apps/functions/isolate/__tmp/package.json'
7 verbose cwd /Users/mac/Projects/suscripciones/packages/api
8 verbose Darwin 20.5.0
9 verbose argv "/Users/mac/.nvm/versions/node/v14.21.3/bin/node" "/Users/mac/.nvm/versions/node/v14.21.3/bin/npm" "pack" "--pack-destination" "/Users/mac/Projects/suscripciones/apps/functions/isolate/__tmp"
10 verbose node v14.21.3
11 verbose npm  v6.14.18
12 error code ENOLOCAL
13 error Could not install from "../../apps/functions/isolate/__tmp" as it does not contain a package.json file.
14 verbose exit [ 1, true ]

isolate.config.json
Screen Shot 2023-06-08 at 1 19 30 PM

pnpm overrides are breaking gcp deployments

I'm trying to deploy to firebase, but it fails if there are pnpm overrides in the root package.json.

ERR_PNPM_LOCKFILE_CONFIG_MISMATCH  Cannot proceed with the frozen installation. The current "overrides" configuration doesn't match the value found in the lockfile

To workaround this, I've duplicated the overrides in each apps package.json, but this does give the warning The field "pnpm" was found in /XXX/apps/my-app/package.json. This will not take effect. You should configure "pnpm" at the root of the workspace instead.

It would be awesome if isolate-package could copy the pnpm.overrides from the root package.json to the isolate/package.json file, so I don't need to manually copy them.

Finally, thanks @0x80 for building isolate-package 🙌 🙌

[BUG] Expected double-quoted property name in JSON at position 283

Hi @0x80, thanks for investigating this problem and creating this package.

I tried to use it with Firebase Functions and a "shared-library" within a PNPM monorepo, but I've stumbled on the following issue when running ISOLATE_CONFIG_LOG_LEVEL=debug npx isolate:

debug Attempting to load config from /<path>/monorepo/firebase-project/functions/isolate.config.json
debug Using configuration: {
  buildDirName: undefined,
  includeDevDependencies: false,
  isolateDirName: 'isolate',
  logLevel: 'info',
  targetPackagePath: undefined,
  tsconfigPath: './tsconfig.json',
  workspacePackages: undefined,
  workspaceRoot: '../..'
}
error SyntaxError: Expected double-quoted property name in JSON at position 283
    at JSON.parse (<anonymous>)
    at readTypedJson (file:///<path>/monorepo/node_modules/.pnpm/[email protected]/node_modules/isolate-package/src/utils/json.ts:14:21)
    at getBuildOutputDir (file:///<path>/monorepo/node_modules/.pnpm/[email protected]/node_modules/isolate-package/src/helpers/get-build-output-dir.ts:21:22)
    at start (file:///<path>/monorepo/node_modules/.pnpm/[email protected]/node_modules/isolate-package/src/index.ts:46:26)

Note: I have installed the packaged under the Firebase functions folder.

Option to preserve package.json scripts

My project has a shared database package that uses Prisma. Prisma generates a client for interacting with your database based on your schema, and the generated files are put into node_modules/@prisma/client. The prisma generate command needs to be run on the target platform. Most guidance is to run this command using a postinstall script within package.json (example).

However, isolate-package omits out scripts from the generated package.json files here, so this approach currently won't work.

Is it possible to add an option to preserve (specific) scripts?

For now I'm working around this with a script that appends the postinstall script to the generated package.json after running isolate.

Thank you for creating this project!

trouble shooting some problems

package '@repo/backend' has conflicts in the following paths:
/Users/kevin/foo/packages/backend
/Users/kevin/foo/apps/gen2/isolate/packages/backend

Struggling with logging/debugging

Do I need to manually delete the isolate folder before each call to isolate?

PNPM used partially and then NPM used to pack...

$ npx isolate
debug Using isolate-package version 1.13.2
debug Found tsconfig at: ./tsconfig.json
debug Workspace root resolved to /Users/kevin/foo
debug Isolate target package (root)/apps/gen2
debug Isolate output directory (root)/apps/gen2/isolate
debug Detected package manager pnpm 8.15.7
debug Detected pnpm packages globs: [ 'apps/*', 'packages/*', 'services/*' ]
debug Registering package ./apps/gen2
debug Registering package ./packages/typescript-config
debug Registering package ./packages/firebase-admin
debug Registering package ./packages/eslint-config
debug Registering package ./packages/connectrpc
debug Registering package ./packages/common
debug Registering package ./packages/backend
debug Using PNPM pack instead of NPM pack
debug Packed (temp)/repo-backend-0.0.0.tgz
debug Packed (temp)/repo-common-0.0.0.tgz
debug Unpacking (temp)/repo-backend-0.0.0.tgz
debug Unpacking (temp)/repo-common-0.0.0.tgz
debug Moved package files to (isolate)/packages/common
debug Moved package files to (isolate)/packages/backend
Error: Command failed: npm pack --pack-destination "/Users/kevin/foo/apps/gen2/isolate/__tmp"

    at ChildProcess.exithandler (node:child_process:422:12)
    at ChildProcess.emit (node:events:517:28)
    at maybeClose (node:internal/child_process:1098:16)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:303:5)

Declared using pnpm but then uses npm pack.

Trying to introduce into firebase monorepo, but need to keep the firebase.json at the root and use codebase for different functions services.

Add support for Rush monorepos

Hi @0x80,

first, thanks a lot for all the time you're spending on this issue.

I have a Rush mono-repo (though I think this shouldn't make a difference) and I'm currently replacing my version of bundling dependencies using your firebase-tools-with-isolate.

I have the following directory structure:
Screenshot from 2024-02-28 15-24-22

The directory packages/apps/backend contains a set of Firebase functions I want to deploy. The content of my firebase.json is the following:

{
"functions": [
  {
    "source": "packages/apps/backend",
    "codebase": "platform-backend",
    "runtime": "nodejs20",
    "ignore": [".rush", "node_modules", "src", ".env*", "*.log", "tsconfig.json", "webpack.config.js"],
    "isolate": true
  }
 ]
}

When running firebase deploy, I get the following error:
Screenshot from 2024-02-28 15-31-17

It seems to me that it's trying to isolate the root directory instead of the backend directory. It should be looking for a tsconfig.json in packages/apps/backend. Is there anything I'm missing?

I tried putting the isolate.config.json file in the backend directory (same error) as well as in the root directory (throws error since there's no package.json in the root dir)

Error: Could not detect language for functions at monorepo/apps/api/isolate

Hello,
I'm trying to setup isolate in my monorepo setup. (turbo + pnpm)

I have root level firebase.json

....
  "functions": [
    {
      "predeploy": ["turbo build --filter api", "isolate"],
      "source": "apps/api/isolate",
      "codebase": "api",
      "runtime": "nodejs18"
    }
  ]
}

And isolate.config.json:

{
  "targetPackagePath": "apps/api",
}

That apps/api have only 1 dependency packages/mypackage and in package.json I'm using "mypackage": "workspace:*".

I checked Prerequisites in the README and made sure that all good.

However when I try to deploy functions from root I got this error:

firebase deploy -P firebase-project --only functions:api:myFunc
...
✔  functions: Finished running predeploy script.
i  functions: preparing codebase api for deployment
i  functions: ensuring required API cloudfunctions.googleapis.com is enabled...
i  functions: ensuring required API cloudbuild.googleapis.com is enabled...
i  artifactregistry: ensuring required API artifactregistry.googleapis.com is enabled...
✔  artifactregistry: required API artifactregistry.googleapis.com is enabled
✔  functions: required API cloudbuild.googleapis.com is enabled
✔  functions: required API cloudfunctions.googleapis.com is enabled

Error: Could not detect language for functions at /path_to_monorepo/apps/api/isolate

And in apps/api/isolate
I see: __tmp/mypackage-0.0.0.tgz

I wonder what I have missed?

Type Error when running isolate

I have a yarn 3.x monorepo with functions as a one of the workspaces located at apps/x-api.
x-api has a local workspace dependency installed from packages/utils.
Trying to just run run isolate results in a type Error in isoalte.

Steps:

  1. install
cd apps/x-api
yarn add isolate-package
  1. run isolate yarn isolate or npx isolate

Error

error TypeError: packagesGlobs.flatMap is not a function
    at createPackagesRegistry (<my project>/node_modules/isolate-package/src/helpers/create-packages-registry.ts:61:6)
    at start (<my project>/node_modules/isolate-package/src/index.ts:80:34)

Adopt packageManager field from root manifest

In a monorepo typically only the root manifest contains a definition for the package manager used. It would be good to copy this field to the target package manifest so that the isolated output adopts the same version.

Add support for passing CLI arguments

Hi @0x80, I wonder if that could be a possibility, I am currently using isolate through cli, but I am not able to pass down any arg.

In case it makes sense, I am willing to help.

Thanks!

Remove excess packages from the list in pnpm lockfile

The list of packages from the root lockfile remains untouched. This does not have have an effect on the number of installed dependencies, but for large projects this can be quite a bit of unnecessary data.

We could run pnpm dedupe to slim it down, or find another way to remove the unwanted packages from the list.

If choosing dedupe it would probably be wise to make it a configurable option, as dedupe does more than just removing the excess packages from the list.

See mono-ts#9

Do not include scripts for internal dependencies

Currently scripts in the package manifests of internal dependencies are included. If they contain a "prepare" script it will get executed on install, which typically triggers a build of that package, which then fails because it requires dev dependencies.

Dependencies should already have been built before isolate happens. All scripts should be discarded probably.

Add flag to omit packageManager field

It seems that cloud-run deployment from source can not (yet) handle the packageManager field correctly. It fails when setting it to pnpm 9.

Add a flag omitPackageManager to not include it in the output manifest.

Google App Engine usage?

Very cool!

I have the same issue where I tried to isolate my local monorepo package manually and bundle them with webpack to deploy an expressjs API using some local monorepo shared libs on Google App Engine

how would I use isolate-package to do the same?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.