atlassian-labs/webpack-deduplication-plugin

Plugin for webpack that de-duplicates transitive dependencies in yarn and webpack-based projects.

Webpack Deduplication Plugin

Plugin for webpack that de-duplicates transitive dependencies in yarn and webpack-based projects.

Usage

Import it from the package

const { WebpackDeduplicationPlugin } = require('webpack-deduplication-plugin');

And add it to your webpack config:

plugins: [
    new WebpackDeduplicationPlugin({
        cacheDir: cacheDirPath,
        rootPath: rootPath,
    }),
]

where:

• cacheDirPath - absolute path to the directory where the cache of the duplicates will be stored. Cache is based on the content of yarn.lock file and will be updated with every change. If not provided then the duplicates will be re-generated with every run.

• rootPath - absolute path to the root of the project. If not provided it will be auto-detected by app-root-path plugin

Development

TBD

Contributions

Contributions to Webpack Deduplication Plugin are welcome! Please see CONTRIBUTING.md for details.

License

Copyright (c) 2020 Atlassian and others. Apache 2.0 licensed, see LICENSE file.

webpack-deduplication-plugin's Issues

Is it possible to support multiple cache folders?

On our projects we're also resolving from extra node_modules paths.

resolve: {
    modules: [anotherNodeModules, 'node_modules']
}

Is it possible somehow to support this scenario?

Add support for package-lock.json

We're not using Yarn, would be nice if we could use the plugin with npm also.

Jest deduplication

Using a "single version" of every package sounds very close to yarn2 pnp. The same resolution algorithm used to power webpack might be used to speedup jest.

Example: https://github.com/arcanis/jest-pnp-resolver

Throwing error with Webpack 5

The plugin is throwing following error with Webpack 5

Error: NormalModuleFactory.beforeResolve is no longer a waterfall hook, but a bailing hook instead. Do not return the passed object, but modify it instead. Returning false will ignore the request and results in no module created.

Removing the return in beforeResolve seems to fix the issue. Not sure if it will break Webpack 4.

diff --git a/node_modules/webpack-deduplication-plugin/build/index.js b/node_modules/webpack-deduplication-plugin/build/index.js
index 083955b..9a62d23 100644
--- a/node_modules/webpack-deduplication-plugin/build/index.js
+++ b/node_modules/webpack-deduplication-plugin/build/index.js
@@ -182,7 +182,7 @@ var WebpackDeduplicationPlugin = /*#__PURE__*/function () {
       var dupVals = Object.values(duplicates);
       compiler.hooks.normalModuleFactory.tap('WebpackDeduplicationPlugin', function (nmf) {
         nmf.hooks.beforeResolve.tap('WebpackDeduplicationPlugin', function (result) {
-          return deduplicate(result, dupVals);
+          deduplicate(result, dupVals);
         });
       });
     }

Does not work in Windows

fast-glob dropped support for backslashes as of 3.0. Since webpack-deduplication-plugin requires fast-glob 3.2.2, this prevents the plugin from working correctly in Windows. No package.json files will ever be globbed, and therefore no duplicates can be identified.

I had to fork the plugin code to fix. Either of the fixes will work:

1. Ensuring globPath is normalized to forward slashes before calling glob.sync
2. Using glob with code as-is instead of fast-glob (with the corresponding perf hit)

Deduplication plugin deduping wrong and breaking the webpack build

ERROR in /buildeng/bamboo-agent-home/xml-data/build-dir/CONFMICRO-CFCPB10727-BUILDBRANCH/node_modules/@atlassian/ptc-embeddable-directory/dist/components/TeamElement/TeamElement.js
build	18-Nov-2020 00:54:13	(@atlassian/confluence-frontend-server) $ yarn run -s bundle Module not found: Error: Can't resolve '/buildeng/bamboo-agent-home/xml-data/build-dir/CONFMICRO-CFCPB10727-BUILDBRANCH/node_modules/@atlaskit/avatar-group/node_modules/@atlaskit/avatar-group/dist/esm/index.js' in '/buildeng/bamboo-agent-home/xml-data/build-dir/CONFMICRO-CFCPB10727-BUILDBRANCH/node_modules/@atlassian/ptc-embeddable-directory/dist/components/TeamElement'

Grep result for the yarn.lock:

mdejongh|~/Projects/Atlassian/confluence-frontend on atlaskit-branch-deploy-AFP-2529-qa-branch
± grep 'avatar-group' yarn.lock
    "@atlaskit/avatar-group" "^8.0.0"
"@atlaskit/avatar-group@^7.0.1":
  resolved "https://packages.atlassian.com/api/npm/npm-remote/@atlaskit/avatar-group/-/avatar-group-7.0.1.tgz#5fde6f9863b9819631cc2903b5d1a3dab4498c25"
"@atlaskit/avatar-group@^8.0.0", "@atlaskit/avatar-group@^8.0.11":
  resolved "https://packages.atlassian.com/api/npm/npm-remote/@atlaskit/avatar-group/-/avatar-group-8.0.11.tgz#09893376b91386aed05c17b82cfcb5548bf694a9"
    "@atlaskit/avatar-group" "^8.0.0"
    "@atlaskit/avatar-group" "^8.0.0"
    "@atlaskit/avatar-group" "^8.0.0"
    "@atlaskit/avatar-group" "^8.0.0"
    "@atlaskit/avatar-group" "^8.0.0"
    "@atlaskit/avatar-group" "^7.0.1"
mdejongh@C02XP02SJGH7|~/Projects/Atlassian/confluence-frontend on atlaskit-branch-deploy-AFP-2529-qa-branch
±

The majority of the app depends on [email protected] but dependency @atlassian/ptc-embeddable-directory wants avatar-group@^7.0.1.