Giter Site home page Giter Site logo

pnpm / pnpm Goto Github PK

View Code? Open in Web Editor NEW
27.4K 140.0 900.0 152.28 MB

Fast, disk space efficient package manager

Home Page: https://pnpm.io

License: MIT License

JavaScript 0.66% TypeScript 99.29% Shell 0.03% Batchfile 0.02%
npm dependency-manager package-manager node nodejs install modules javascript

pnpm's Introduction

简体中文 | 日本語 | 한국어 | Italiano | Português Brasileiro

Fast, disk space efficient package manager:

  • Fast. Up to 2x faster than the alternatives (see benchmark).
  • Efficient. Files inside node_modules are linked from a single content-addressable storage.
  • Great for monorepos.
  • Strict. A package can access only dependencies that are specified in its package.json.
  • Deterministic. Has a lockfile called pnpm-lock.yaml.
  • Works as a Node.js version manager. See pnpm env use.
  • Works everywhere. Supports Windows, Linux, and macOS.
  • Battle-tested. Used in production by teams of all sizes since 2016.
  • See the full feature comparison with npm and Yarn.

To quote the Rush team:

Microsoft uses pnpm in Rush repos with hundreds of projects and hundreds of PRs per day, and we’ve found it to be very fast and reliable.

npm version Join the chat at Discord OpenCollective OpenCollective Twitter Follow Stand With Ukraine

Platinum Sponsors

Gold Sponsors

Silver Sponsors

Support this project by becoming a sponsor.

Background

pnpm uses a content-addressable filesystem to store all files from all module directories on a disk. When using npm, if you have 100 projects using lodash, you will have 100 copies of lodash on disk. With pnpm, lodash will be stored in a content-addressable storage, so:

  1. If you depend on different versions of lodash, only the files that differ are added to the store. If lodash has 100 files, and a new version has a change only in one of those files, pnpm update will only add 1 new file to the storage.
  2. All the files are saved in a single place on the disk. When packages are installed, their files are linked from that single place consuming no additional disk space. Linking is performed using either hard-links or reflinks (copy-on-write).

As a result, you save gigabytes of space on your disk and you have a lot faster installations! If you'd like more details about the unique node_modules structure that pnpm creates and why it works fine with the Node.js ecosystem, read this small article: Flat node_modules is not the only way.

💖 Like this project? Let people know with a tweet

Getting Started

Benchmark

pnpm is up to 2x faster than npm and Yarn classic. See all benchmarks here.

Benchmarks on an app with lots of dependencies:

Usage Trend

Usage Trend of pnpm

image

Backers

Thank you to all our backers! Become a backer

Contributors

This project exists thanks to all the people who contribute. Contribute.

License

MIT

pnpm's People

Contributors

andreypopp avatar await-ovo avatar btea avatar chengcyber avatar d3lm avatar davej avatar dev-itsheng avatar etamponi avatar exe-boss avatar fireairforce avatar gluxon avatar greenkeeper[bot] avatar greenkeeperio-bot avatar illright avatar jakebailey avatar jonathanmorley avatar juanpicado avatar kenrick95 avatar ksxgithub avatar lvqq avatar mcmxcdev avatar nachoaldamav avatar renovate-bot avatar renovate[bot] avatar rstacruz avatar stevenpetryk avatar umireon avatar yaodingyd avatar zkochan avatar zxbodya avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pnpm's Issues

getaddrinfo ENOTFOUND registry.npmjs.org registry.npmjs.org:443

When I executed pnpm install I've got random error like

getaddrinfo ENOTFOUND registry.npmjs.org registry.npmjs.org:443
GET registry.npmjs.org/mime-types/~2.1.8

But if I run curl https://registry.npmjs.org/mime-types/~2.1.8 I see expected json output.
With each command run library changes.

`standard` doesn't work

standard has this structure:

.
'- standard
   |- standard-engine
   |  '- eslint
   '- eslint-config-standard-react
      '- eslint-plugin-react

unfortunately, eslint now can't see eslint-config-standard-react in our setup because of the store format:

.
'- node_modules
  |- .store
  |  |- standard
  |  |- standard-engine
  |  |- eslint
  |  |- eslint-config-standard-react
  |  '- eslint-plugin-react
  '- standard

there are ways around this... and the new 0.12 store format possibly paves the way to make that possible.

also, the npm3-style flat repo might also work here.

Don't fail on `pnpm install` without a package.json

Doing pnpm install in a path without package.json shows a cryptic error message and exits with code 1.

~ $ cd /tmp
/tmp $ pnpm i

 ! Cannot read property 'dependencies' of undefined

For comparison, npm is also a little strange in its behavior in this case, but it at least exits successfully:

/tmp $ npm i
npm WARN enoent ENOENT: no such file or directory, open '/private/tmp/package.json'
npm WARN tmp No description
npm WARN tmp No repository field.
npm WARN tmp No README data
npm WARN tmp No license field.

# node_modules will be enumerated here, if it exists

Related #47.

Use registry url configured in npm

This will be useful for local npm mirrors like sinopia, and could lead to significant speed boost in downloads.

> npm config get registry
http://localhost:4873/

Shrinkwrap

I was pretty excited to hear about this project. I can see that shrinkwrap is not supported at the moment but do you plan to support it at later point? And if so, what's the schedule?

lock node_modules when running

you can do two pnpm install's at the same time at the same project, though it will probably break things along the way. a lockfile will solve this.

Where is help wanted?

Rather than jump into a random feature, are there any discrete chunks of functionality that you would appreciate PRs for?

Use caching to minimise bandwidth usage / improve performance

To see whether it's worth exploring to achieve performance improvements, I quickly implemented something which would use the tarball from the .npm cache if present (and fall back to fetching from the server if not). It only improved the install time by about 7% 😞 (for quite a big project)

However, I then tried (something a little hackier) to force the package resolver to use npm's cached metadata if present (falling back to making a request to the registry if not). This improved the speed significantly:

Current master - no cache:

35.86s user 17.93s system 83% cpu 1:04.81 total

...cache

24.04s user 11.90s system 94% cpu 38.115 total

(BTW there does appear to be a slight performance regression in today's release - at the very end it's taking longer than before)

Getting this weird error

Anyone know how to fix this:

fs.js:984
binding.chown(pathModule._makeLong(path), uid, gid, req);
^
TypeError: uid must be an unsigned int
at TypeError (native)
at fs.chown (fs.js:984:11)
at /usr/local/lib/node_modules/pnpm/node_modules/tar-fs/index.js:215:7
at FSReqWrap.oncomplete (fs.js:95:15)

getaddrinfo ENOTFOUND registry.npmjs.org

Still getting this with 0.17 on different packages each time.

gulp ^3.8.11                                                      ERROR ✗    
...
getaddrinfo ENOTFOUND registry.npmjs.org registry.npmjs.org:80
GET registry.npmjs.org/deprecated/-/deprecated-0.0.1.tgz

Global store

It'd be nice for the store to be in ~/.pnpm/store.

The store is currently defined here:

https://github.com/rstacruz/pnpm/blob/20df53b3193d0b548a21d7287e0f9af3b62f8143/bin/pnpm-install#L93

But I'd like to have some features here:

  • use rc to store this path in .pnpmrc
  • Probably want to use homedir (or similar) to resolve ~'s
  • relSymlink() is currently being used to make symlinks from .store/[email protected] => xxx, automatically figuring out the relative path to do so. However, it should be amended such that if the store path is NOT inside the package dir (ie, it's in home), the symlink will be absolute, not relative
  • ...whatever edge cases I haven't thought of

Use registry-url

https://www.npmjs.com/package/registry-url

var registryUrl = require('registry-url')

function registryUrlFor (package) {
  if (package.substr(0, 1) === '@') {
    return registryUrl(package.split('/')[0])
  } else {
    return registryUrl()
  }
}

registryUrlFor('@foo/bar') //=> registryUrl('@foo') => 'https://registry.foo.io'

This will be useful for private scoped modules support. See #9 @indexzero

Possibly helpful?

Great work with this!

I don't know if it will help you in any way, but if you see any code in https://github.com/dtrejo/urn that you'd like to "steal" / re-use, please feel free. It does "cp -R" to install packages from ~/.npm when there is no internet.

Cheers and keep up the good work!
David

Note: just looked at the code; probably nothing you'll want to use. Thanks for pnpm!

--global

npm install -g bower

Don't know how to do this yet... need to investigate.

npm prefix -g will show you where node modules are installed globally. Should pnpm dump things into $(npm prefix -g)/lib/node_modules? what about Windows?

How can npm prefix -g be obtained programatically?

ied dumps your globals into ~/.node_modules, I believe. Not sure on what the pros and cons of that are at the moment, though.

Different behavior with npm install

If I run npm install in dir node_modules, npm will not create a new node_modules, it'll just put the installed lib under the same node_modules. but pnpm will create a new node_modules, I'm not sure if this is expected.

option to for .store location

node_modules/.store makes sense by default, but there should be an option to put it wherever you want. my main thought is that i'd like to be able to rm -rf node_modules. if i don't want to rm -rf the store, then i would place it outside the node_modules directory.

Intermittent ENOENT errors during symlinking

I'm getting a lot of this type of error since getting the latest version...
It looks like a timing thing because it doesn't happen consistently - i.e. not always and when it does with different packages. Always a similar stack trace though.

 ! ENOENT: no such file or directory, open 'my-app/node_modules/.store/[email protected]/_/package.json'
Error: ENOENT: no such file or directory, open '/my-app/node_modules/.store/[email protected]/_/package.json'
    at Error (native)
    at Object.fs.openSync (fs.js:584:18)
    at Object.fs.readFileSync (fs.js:431:33)
    at requireJson (/src/pnpm/lib/fs/require_json.js:10:42)
    at symlinkToModules (/src/pnpm/lib/install.js:274:17)
    at /src/pnpm/lib/install.js:97:20
From previous event:
    at /src/pnpm/lib/install.js:97:10
From previous event:
    at install (/src/pnpm/lib/install.js:89:6)
    at /src/pnpm/lib/install_multiple.js:14:12
    at Array.map (native)
    at installMultiple (/src/pnpm/lib/install_multiple.js:13:27)
    at /src/pnpm/lib/install.js:222:16
From previous event:
    at buildInStore (/src/pnpm/lib/install.js:222:6)
    at /src/pnpm/lib/install.js:179:20
    at processImmediate [as _immediateCallback] (timers.js:383:17)

--userconfig

After digging, this behavior is already possible by just using --config instead of --userconfig because of this line of rc (used by registry-url). Should this difference be noted in the readme, or for the sake of parity should this be brought up as an issue on rc?

uid must be an unsigned int

pnpm is awesome.. but I'm having issue on installing.. please help.

Issue:
binding.chown(pathModule._makeLong(path), uid, gid, req);
^

TypeError: uid must be an unsigned int
at TypeError (native)
at fs.chown (fs.js:1082:11)
at /usr/local/lib/node_modules/pnpm.js/node_modules/tar-fs/index.js:215:7
at FSReqWrap.oncomplete (fs.js:83:15)

Add one more layer in .store for prod vs dev (vs others?)

This makes it possible to add a single dev (or prod) directory to a .gitignore, or a .dockerignore, or similar.

Currently with the flat hierarchy there's no delineation between prod and dev deps (same problem in npm3 tbh: npm/npm#9674 )

Deduping will perhaps be more difficult, so I get this is a big ask.

--save

npm install --save express

pnpm doesn't appear to be able to find certain packages

When I run pnpm i on a project with plenty of dependenices, it'll install a few packages and then get to a point where it throws:

getaddrinfo ENOTFOUND registry.npmjs.org registry.npmjs.org:443
GET registry.npmjs.org/shebang-regex/%5E1.0.0

except the address on the second line changes, of course, but when I run the command again, it continues to install more packages and fails on another package - never the same one.

Reimplement bin/pnpm to .js

It's currently a bash script. I thought this'd speed things up... but I'm not sure it's necessary, plus it breaks Windows compatibility.

Install from files

npm install file:../foo/bar

This is a little easier.... I think... I'm not sure. First I need to investigate how npm does this. How does it handle .gitignore? .npmignore?

I think the way to do this is not to do cp -R, but to tarball the file: path (with proper paths excluded), and untar it at the destination directory.

Idea: Add node version to the store hash computation

node_modules/.store/[email protected]

This is how modules are stored. However, once you switch node versions (eg, from node 4.2.4 to 5.3.0), you'll be greeted by this:

> require('node-sass')
Error: The `libsass` binding was not found in /xxx/node_modules/.store/[email protected]/vendor/darwin-x64-47/binding.node
This usually happens because your node version has changed.

This sucks. To mitigate this, compiled modules should have the node major version appended to their store ID:

node_modules/.store/[email protected]@node4

While not a complete solution, this allows you to simply run pnpm install to rebuild modules as needed.

In addition to this, there should be integrity checks that symlinks are linking to the correct versions... I guess.

Add 'unmet dependency' warnings

npm has these:

npm WARN unmet dependency /home/travis/build/rstacruz/pnpm/node_modules/nixt/node_modules/changelog requires chalk@'^0.5.1' but will load
npm WARN unmet dependency /home/travis/build/rstacruz/pnpm/node_modules/chalk,
npm WARN unmet dependency which is version 1.1.1
npm WARN unmet dependency /home/travis/build/rstacruz/pnpm/node_modules/nixt/node_modules/changelog requires semver@'^4.0.3' but will load
npm WARN unmet dependency /home/travis/build/rstacruz/pnpm/node_modules/semver,
npm WARN unmet dependency which is version 5.1.0

pnpm's model isn't immune to this (in fact is more prone to it), so it might be prudent to have it implemented as well.

Install from GitHub

npm install rstacruz/scourjs

Okay, this is a tough one. I don't even know where to begin describing this... sorry, will come back to it later

Benchmark suite

I would be up for working on this.

My thinking is that I will mock the network endpoints (registry mocker here: https://github.com/npm/npm-registry-mock) so that it's not network dependent. Could also be integrated with Travis, although I'm guessing that the performance might differ greatly based upon the VM you get assigned when the build is run.

@rstacruz: Do you have any thoughts?

Implementing scoped modules

As a new developer to the code base I was hoping you could give some foot notes on where I should start looking if I wanted to implement scoped module support.

support inherited dependencies ("peerDependencies")

packages like pg-then will require('pg') even though they don't declare it as a dependency. This works with npm since pg-then ends up being installed in the same node_modules folder as pg but when they are symlinked from a global store this doesn't work. And babel-runtime does require('babel-runtime'). haha

These packages should declare these dependencies in "peerDependencies" but they don't because with npm they can get away without it.

And BTW the hackery I mentioned in #1 won't be necessary if pnpm support "peerDependencies" and if packages use it correctly

`pnpm ls`

While it seems superficial, being able to do an analog of npm ls is pretty crucial.

I'll eventually want to have an analog of npm dedupe, which'd allow certain modules that assume a flat dependency tree to work (like standard, see #5). To do this, we need generate a dependency tree.

pnpm install --production

From npm help install:

With the --production flag (or when the NODE_ENV environment variable is set to production), npm will not install modules listed in devDependencies.

binstubs of deep dependencies

When you do this:

npm install node-sass
find node_modules/.bin

npm2 and pnpm will get you the node-sass binstub:

node_modules/.bin
node_modules/.bin/node-sass

but npm3, with its flat dependency tree model, gets you binstubs of deep dendencies:

node_modules/.bin
node_modules/.bin/har-validator
node_modules/.bin/mkdirp
node_modules/.bin/node-gyp
node_modules/.bin/node-sass
node_modules/.bin/nopt
node_modules/.bin/rimraf
node_modules/.bin/sassgraph
node_modules/.bin/semver
node_modules/.bin/sshpk-conv
node_modules/.bin/sshpk-sign
node_modules/.bin/sshpk-verify
node_modules/.bin/strip-indent
node_modules/.bin/uuid
node_modules/.bin/which
node_modules/.bin/window-size

I'm planning on not supporting npm3's model on this, and keeping pnpm / npm2's behavior. While it's possible to support this in pnpm, I personally don't see the benefit of npm3's model and would rather avoid complexity that will not be useful.

Any objections? (cc @davej @misterbyrne @indexzero and others who may be lurking)

Module name on npm

The current module name pnpm.js is a bit funky. I wonder would @crcn be interested in releasing the pnpm name?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.