alexisvincent/systemjs-config-builder

Generate SystemJS config files from node_modules

SystemJS Config Builder

Generate SystemJS config files from node_modules

Overview

The easiest way to include modules in SystemJS is currently through JSPM. This project aims to provide an alternative, by leveraging the deterministic installs of Yarn (also works with npm) and generating config files to explain to SystemJS how to resolve modules from node_modules.

Roadmap

• Trace node_modules and build registry
• Generage compatability configs for packages
• Generate valid SystemJS config file
• Make sure loading React.JS works
• Cache registry (to speed up subsequent generations)
• Use own @node polyfills. See #1 for progress
• Respect jspmPackage: true
• Perform path optimisations (nm: -> node_modules), for smaller configs
• Fully support npm resolve algorithym (upper node_module and single dep node_modules)
• Allow simple local overrides
• Use JSPM overrides
• Allow dependency filtering (so we don't include deps meant for just the server)
• Provide CLI flag to optimize generated.config.js
• Allow configurability
• Make more robust

Currently, given this package.json, SystemJS Config Builder generates this config.

Usage

Example Project (with usage instructions)

You can test the generation via the cli in systemjs-tools.

yarn global add systemjs-tools

add SystemJS @node browser polyfills

yarn add systemjs-nodelibs

and generate the config

systemjs config

optionally you can also add a postinstall hook to do this automatically

package.json

{
  "scripts": {
    "postinstall": "systemjs config"
  }
}

Thanks

Thanks to Juha Järvi for helping me flesh out this idea and for the discussions regarding his awesome cbuild project.

systemjs-config-builder's Issues

yarn.lock file

Just wanted to let you know I'm doing some experimenting over here with using the yarn.lock file as we originally discussed. Not sure if it will be useful but at least I am learning a lot :-). I would really like to make the system config a pure function of the lockfile. But I discovered that is not possible as we need to add package config from from the jspm-node conversion stuff. This part is also quite slow, would be nice to find a way to make it faster. Anyway, the system config is now a pure function of the lockfile and the extra jspm-config.

I'm basically doing the same thing as in this repo, just divided the steps a little different to get info from the lockfile rather than the node_modules folder. I was thinking we could add some code that looks for a yarn.lock file and uses it if present, otherwise falls back to scanning through node_modules. Maybe I could contribute something like this but we would have to re-arrange a bit to make that work. And as I said, not sure if it will be useful as the slow part is probably not scanning node_modules.

One thing I noticed is that I did not have to hard-code the node-libs packages using the short name in the map. It worked with the long names. What I mean is that this worked:

SystemJS.config({
  "paths": {
    "nm:": "node_modules/"
  },
  "map": {
    "jspm-nodelibs-assert": "nm:jspm-nodelibs-assert",
    "jspm-nodelibs-buffer": "nm:jspm-nodelibs-buffer",
    "jspm-nodelibs-child_process": "nm:jspm-nodelibs-child_process",
    "jspm-nodelibs-cluster": "nm:jspm-nodelibs-cluster",
 ...

If I understood correctly the code in this repo will map with the short names like "assert", "buffer" and so on.

Properly deal with jspmPackage: true

I assumed this was done in jspm/npm, but it isn't

Doesn't seem to work?

I follow the example to do:

// in an empty folder.
yarn global add systemjs-tools
yarn add systemjs systemjs-nodelibs systemjs-plugin-babel react react-dom
systemjs config

But the systemjs config just exit and no generated.config.js is created.

NPM shrinkwrap yarn.lock as source config?

Hi Guys,

Wondering your thoughts on this ...

I can see an usecase for generating the systemjs from an npm shrinkwrap file (or yarn.lock file) and not by walking the node_modules dependencies. If you want a consistently repeatable systemjs config file then - it's a better option than trusting developers (or C.I environments) node_modules installs.

When people use optimistic semver in node modules, it can obviously have the side effect that different devs are developing on the same NPM module, but with a different set of downstream dependencies. This can be a disaster for sane release management.

If the systemjs config file was produced from a stable dependency tree that can be versioned, e.g. yarn.lock or npm shrinkwrap - then I could see this been more useful.

Excluding modules

I saw there's an item on the checklist in the readme about this. How about (optionally?) excluding all devDependencies?

Scoped packages are not added to config

In case a package is published under a scope, it is not added to the generated config. An example for this are the Angular >= 2.0.0 packages that are published under the @angular scope.
The resulting node_modules directory structure looks like this:

node_modules
├── @angular
│   ├── common
│   │   ├── README.md
│   │   ├── bundles
│   │   ├── index.d.ts
│   │   ├── index.js
│   │   ├── index.js.map
│   │   ├── index.metadata.json
│   │   ├── package.json
│   │   ├── src
│   │   └── testing
│   ├── compiler
│   │   ├── README.md
│   │   ├── bundles
│   │   ├── index.d.ts
│   │   ├── index.js
│   │   ├── index.js.map
│   │   ├── index.metadata.json
│   │   ├── package.json
│   │   ├── src
│   │   └── testing
│   ├── core
│   │   ├── README.md
│   │   ├── bundles
│   │   ├── index.d.ts
│   │   ├── index.js
│   │   ├── index.js.map
│   │   ├── index.metadata.json
│   │   ├── package.json
│   │   ├── src
│   │   └── testing
│   ├── platform-browser
│   │   ├── README.md
│   │   ├── bundles
│   │   ├── index.d.ts
│   │   ├── index.js
│   │   ├── index.js.map
│   │   ├── index.metadata.json
│   │   ├── package.json
│   │   ├── src
│   │   └── testing
│   └── platform-browser-dynamic
│       ├── README.md
│       ├── bundles
│       ├── index.d.ts
│       ├── index.js
│       ├── index.js.map
│       ├── index.metadata.json
│       ├── package.json
│       ├── src
│       └── testing
...

Investigate why this doesn't work with yarn

Installing via npm works. Yarn does not.

The problem is that Yarn installs the module with thispackage.json

{
  "name": "string_decoder",
  "version": "0.10.31",
  "description": "The string_decoder module from Node core",
  "main": "index.js",
  "dependencies": {},
  "devDependencies": {
    "tap": "~0.4.8"
  },
  "scripts": {
    "test": "tap test/simple/*.js"
  },
  "repository": {
    "type": "git",
    "url": "git://github.com/rvagg/string_decoder.git"
  },
  "homepage": "https://github.com/rvagg/string_decoder",
  "keywords": [
    "string",
    "decoder",
    "browser",
    "browserify"
  ],
  "license": "MIT"
}

to node_modules/string_decoder-browserify instead of node_modules/string_decoder.

This is different to the npm behaviour, which works as expected.

@thejameskyle maybe you can help with this?

to reproduce,

npm install
cd test
npm install or yarn install
jspm install
npm run config

the above breaks if you installed via yarn

Unhandled rejection TypeError: Cannot read property 'name' of null

I followed the brief installation directions:

1. yarn global add systemjs-tools
2. yarn add systemjs-nodelibs
3. systemjs config

Then I got:

Unhandled rejection TypeError: Cannot read property 'name' of null
    at ~/.config/yarn/global/node_modules/systemjs-config-builder/dist/index.js:126:42
    at Array.forEach (native)
    at ~/.config/yarn/global/node_modules/systemjs-config-builder/dist/index.js:125:25
    at tryCatcher (~/.config/yarn/global/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (~/.config/yarn/global/node_modules/bluebird/js/release/promise.js:510:31)
    at Promise._settlePromise (~/.config/yarn/global/node_modules/bluebird/js/release/promise.js:567:18)
    at Promise._settlePromise0 (~/.config/yarn/global/node_modules/bluebird/js/release/promise.js:612:10)
    at Promise._settlePromises (~/.config/yarn/global/node_modules/bluebird/js/release/promise.js:691:18)
    at Promise._fulfill (~/.config/yarn/global/node_modules/bluebird/js/release/promise.js:636:18)
    at Promise._settlePromise (~/.config/yarn/global/node_modules/bluebird/js/release/promise.js:580:21)
    at Promise._settlePromise0 (~/.config/yarn/global/node_modules/bluebird/js/release/promise.js:612:10)
    at Promise._settlePromises (~/.config/yarn/global/node_modules/bluebird/js/release/promise.js:691:18)
    at Promise._fulfill (~/.config/yarn/global/node_modules/bluebird/js/release/promise.js:636:18)
    at Promise._resolveCallback (~/.config/yarn/global/node_modules/bluebird/js/release/promise.js:431:57)
    at Promise._settlePromiseFromHandler (~/.config/yarn/global/node_modules/bluebird/js/release/promise.js:522:17)
    at Promise._settlePromise (~/.config/yarn/global/node_modules/bluebird/js/release/promise.js:567:18)

Derive own @node polyfills

At the moment, everything works if you use jspm to install the @node polyfills. But this still requires you to have JSPM installed.

Normally we would just be able to install the jspm-nodelibs-* packages, but these don't follow the npm conventions. For example, jspm-nodelibs-buffer has the following package.json:

{
  "main": "buffer.js",
  "name": "jspm-nodelibs-buffer",
  "version": "0.2.0",
  "registry": "github",
  "map": {
    "./buffer.js": {
      "node": "@node/buffer",
      "browser": "buffer-browserify"
    }
  },
  "dependencies": {
    "buffer-browserify": "npm:buffer@^4.0.0"
  },
  "jspmNodeConversion": false
}

Notice the dependency mapping. This causes npm to fail and yarn to do weird things. So I need to create forks of all these libs, and ideally make this automated so that whenever the underlying package changes, we just rebuild.

Slashes on windows

In the generated.config.js file, all the module locations (in node_modules) are with escaped back slashes. The imports in the code themselves, though, use forward slashes. This causes at least one issue I noticed with systemjs, resolving packages for an import, since the backslashes don't match. (See https://github.com/systemjs/systemjs/blob/master/dist/system.src.js#L2205. the substr check will always be false because the slashes are different)

The natural solution seems to be normalizing all backslashes to be forward slashes.

Provide a LICENSE file

Hey @alexisvincent nice libray! Should be nice if the repo has LICENSE file

AMD modules assigned "format": "cjs"

The systemjs plugin-json package, for example, uses define but gets "format": "cjs".

out of memory for large dependency graphs

Hey,

Ive been running out of memory when trying to generate System config for large modules. Increasing the Node max mem is not really helping. Problem seems to be in JSPM-NPM.

Could I contribute a enhancement that only scans "dependencies" and "peerDependencies" and skips "devDependencies" ?? Will really cut down the tree walking!

I guess some command line arg could be used to determine when to ignore devDepends.

Interested?

Systemjs config do nothing

I am did steps from guide, but command do nothing. Its works?

Does not work with plugin-typescript

When trying to use plugin-typescript the chrome tab hangs with 100% CPU usage.

How it works (documentation)

Good work on this package so far!

I have breifly looked through the code and are trying to understand the details of how it works. In doing so I thought I would write my findings down and maybe it could be added to a "How it works" section of the readme in the future. It may also help in discussion of improvements to the algoritm. So here it is, please let me know what I have misunderstood :-).

How it works

SystemJS needs three pieces of information to load a package:

P1. Extended package.json information (eg. format: cjs).
P2. The URL to the package root.
P3. Map to dependent packages.

systemjs-config-builder provides the above information for each package by this algoritm:

A1. Walk node_modules recursively and look for package.json. Build a registry (map) of packages with key name@version and value {location}. The recursive part ensures that packages that has private dependencies in their own node_modules folder are included in the registry.

A2. Iterate the registry and call jspm to generate the extended package.json information needed (P1). The registry values now have {location, config}.

A3. Loop through the registry and write information to a generated.config.js file.

A3.1 Output path section (P2). This section is hardcoded to this:

"paths": {
    "nm:": "node_modules/"
  }

A3.2 Output map section. For each package (1...N) set name to registry key minus @version part and output:

"map": {
    "[name1]": "nm:[name1]",
    ...
    "[nameN]": "nm:[nameN]",
}

A3.3 Output packages section. For each package (1...N) set name to registry key minus @version part, and config from the registry value and output:

  "packages": {
    "nm:[name1]": {
      [config1]
    },
    ...
    "nm:[nameN]": {
      [configN]
    }
}