Updating shrinkpacked dependencies

[jamiemason]

Updating shrinkpacked dependencies

[markerikson]

I wrote up some instructions for this as part of my sample React/Redux/Webpack project config repo, at https://github.com/markerikson/react-redux-cesium-testing-demo . Pasting those instructions here for reference:

Managing Dependencies using Shrinkpack

The recent uproar over the unpublishing of "left-pad" and the subsequent breakage of thousands of NPM packages and build environments demonstrates the need to maintain a fixed list of all transitive dependencies. However, the common suggestion of checking in /node_modules/ is a bad idea. A typical /node_modules/ folder is easily 20K files and 150MB, and probably has platform-specific post-installation build artifacts (such as node-sass's native-code SASS compiler).

While researching this issue in 2015, I ran across a tool called https://github.com/JamieMason/shrinkpack , which seems to solve most of this issue. It simply refers to a npm-shrinkwrap.json file, and uses NPM's caching abilities to grab the tarballs for each exact dependency. It copies those to a /node_shrinkwrap/ folder, which can then be easily committed. That folder will probably be more like 500-750 files and 20MB - much more manageable. Re-running shrinkpack after a package update will just add or remove a few tarballs as needed, rather than having to re-commit hundreds or thousands of files under /node_modules/. In addition, having the pre-install packages available means that the project can have npm install run on multiple platforms, without worrying that a Linux build artifact got checked out on Windows. There's also no need for NPM to go out to a server during an install, as all dependencies are right here and pre-specified. That means a build environment doesn't have to have a network connection.

A typical Shrinkpack workflow for managing dependencies looks like this:

# one-time global install of Shrinkpack
npm install -g shrinkpack

# install whatever packages you want to update
npm install some-package --save-dev

# once you are ready to persist the upgrade, then re-generate 
# the npm-shrinkwrap.json file, including devDependencies
npm shrinkwrap --dev  

# re-run Shrinkpack to copy the tarballs to the ./node_shrinkwrap/, folder, 
# and rewrite the shrinkwrap links to point to ./node_shrinkwrap/some-tarball.tgz
shrinkpack

# Stage updated shrinkwrap file
git add npm-shrinkwrap.json
# Stage added and removed tarballs
git add -A node_shrinkwrap
git commit -m "Updated some-package"

[DrewML]

One small note on the write-up above (although it is awesome and super well done): Avoid installing shrinkwrap globally if you are working on the project with others. This was the source of some major problems Grunt had during it's early days (and why tools like grunt/gulp/et al. have a separate CLI package that just invokes the version local to the project).

If shrinkpack introduces any breaking changes, and you and your teammates are using a different version, it will lead to lots of weirdness.

As a recent example, shrinkpack switched from using gzipped tarballs to uncompressed tarballs. If you were to have an old version of shrinkpack installed globally, but your teammate had the latest, you'd be swapping back and forth between compressed and uncompressed tarballs, which would be a bit annoying.

You can install shrinkpack in your project using npm install shrinkpack --dev, setup an npm run script for npm shrinkwrap --dev && shrinkpack. This will avoid any issues that could arise due to version mismatches between committers.

[IanVS]

Thanks @markerikson and @DrewML. As a part of the ESLint and eslint atom plugin teams, I am well aware of the problems that global vs. local installations can have. :-p

I've made an npm run script to shrinkwrap and then shrinkpack, and it works great. 👍

[JamieMason]

Thanks everyone, I still need to finish the README but will get to it soon 👍

[JamieMason]

Please see the updated README and please feel free to offer any suggestions for additions or changes.

A lot of areas are quite wordy and also I've struggled to articulate what the value of shrinkpack is, so any ideas around that would be helpful – a lot of the current talk is about risk mitigation and resilience, but most don't seem to be too interested in that, maybe install speed is more of a draw.

[IanVS]

I noticed my install speed went down when I started using shrinkpack, and for me the value really is risk mitigation and not relying on npm registry for deploying my code. So, I wouldn't promise faster installs necessarily.

Note: I was already keeping the npm local cache between deployments.

[JamieMason]

I noticed my install speed went down when I started using shrinkpack

Yikes, do you have any more info on this please? I'll try to improve it.

[IanVS]

Well, since I was already keeping the local npm cache and using shrinkwrap, the only thing that really changed when I started using shrinkpack was that npm doesn't need to make the http requests to check the version on the registry. Those were super fast usually, so that doesn't save a lot of time. I'm not familiar with the inner workings of npm, so I'm not sure why the speed dropped (was only about 30 seconds). I also haven't dug very deeply into what part of the process is taking longer, because I can live with the speed penalty for the increased safety. If I get some time I'll try to do some more investigation and if I can confirm and narrow it down, I'll open an issue.

[JamieMason]

Thanks a lot @IanVS, appreciate it. I will take a look too.