snyk / nodejs-lockfile-parser Goto Github PK

Generate a Snyk dependency tree from package-lock.json or yarn.lock file

License: Other

TypeScript 99.86% JavaScript 0.14%

snyk snyk-io node cli-plugin

nodejs-lockfile-parser's Introduction

Snyk helps you find, fix and monitor for known vulnerabilities in your dependencies, both on an ad hoc basis and as part of your CI (Build) system.

Snyk Node.js Lockfile Parser

This is a small utility package that parses lock file and returns either a dependency tree or a dependency graph. Dependency graphs are the more modern data type and we plan to migrate fully over.

Dep graph generation supported for:

package-lock.json (at Versions 2 and 3)
yarn.lock
pnpm-lock.yaml (lockfileVersion 5.x, 6.x and 9.x)

Legacy dep tree supported for:

package-lock.json
yarn 1 yarn.lock
yarn 2 yarn.lock

nodejs-lockfile-parser's People

Contributors

Stargazers

Watchers

nodejs-lockfile-parser's Issues

https://github.com/OpenSourceOrg/2021-members-campaign/blob/smaffulli-patch-1/static/js/components/member-list/members.json

https://github.com/OpenSourceOrg/2021-members-campaign/blob/smaffulli-patch-1/static/js/form.js

Improve testing code and introduce snapshots

The testing code right now is spread between node-tap and jest. Moreover, the process of introducing properties and updating fixtures is quite cumbersome and not for everyone.

This could be improved, in order to make the testing seamless and smoother. I propose following changes.

Migration of all tests to Jest
Use of snapshotting for fixture expected dependency trees comparison, it would give us better performance along with easier management and updating
Further modularization of the test code

npm lock version evaluation is incorrectly reporting an error

I have a nx monorepo project where I use snyk-cli snyk monitor with --all-projects flag to and it uses this library to identify the lock version. snyk-cli version 1.1072.0 and earlier which uses nodejs-lockfile-parser 1.44.0 works fine without issues, but 1.1073.0 onwards which relies on nodejs-lockfile-parser 1.45.1 fails for me with the below error .

Unsupported npm lockfile version in package-lock.json. Please provide a [package-lock.json](<url-removed>) with lockfileVersion 1, 2 or 3

Note: There are no issues when I do a snyk test and it only occurs when I do a snyk monitor.
Here is a comparison of snyk cli release 1.1073.0 snyk/cli@v1.1072.0...v1.1073.0

Here is a comparison of nodejs-lockfile-parser v1.44.0...v1.45.1

The above shows that lib/util.ts was added; it is a brand-new file and here the npm lock version is evaluated. The error message I see in our pipeline comes from this file.

I'm unsure at this point if the issue is on the snyk cli end or this library, but have narrowed down when the issue was introduced.

I have opened a snyk support ticket, and adding this here in case a contributor on this project can help.

Zendesk test

Nara

https://raw.githubusercontent.com/Pawdroid/Free-servers/main/sub

bug: no OutOfSyncError on github revision mismatch

reproduce

cd $(mktemp -d)
npm init -y
npm i snyk-nodejs-lockfile-parser
npm i "github:milahu/postcss-nodegui-autoprefixer#e180d6a5f2f313d634f73637a285c129de90d530"

sed -i 's/e180d6a5f2f313d634f73637a285c129de90d530/f4311b3ce656395d469e9a7df0b940bdc184a757/' package.json
# now package.json and package-lock.json are out of sync

src="const read = path => require('fs').readFileSync(path, 'utf8');"
src+="async function main() { console.dir(await require('snyk-nodejs-lockfile-parser')."
# call buildDepTree
# last argument: strictOutOfSync = true
src+="buildDepTree(read('package.json'), read('package-lock.json'), true, 'npm', true)"
src+="); }; main()"
node -e "$src"

expected result: should throw OutOfSyncError

actual result

{
  dependencies: {
    'postcss-nodegui-autoprefixer': {
      labels: [Object],
      name: 'postcss-nodegui-autoprefixer',
      version: 'git+ssh://[email protected]/milahu/postcss-nodegui-autoprefixer.git#e180d6a5f2f313d634f73637a285c129de90d530',
      dependencies: [Object]
    },
    'snyk-nodejs-lockfile-parser': {
      labels: [Object],
      name: 'snyk-nodejs-lockfile-parser',
      version: '1.37.0',
      dependencies: [Object]
    }
  },
  hasDevDependencies: false,
  name: 'tmp.ibogyemwlr',
  size: 300,
  version: '1.0.0',
  meta: { lockfileVersion: 2, packageManager: 'npm' }
}

fix: parse child peerDependencies in package-lock.json

child peerDependencies should be parsed

consider svelte-nodegui-starter

part of the dependency tree

svelte-nodegui-starter
+ [email protected]
+ + [email protected]
+ + [email protected]

acorn-import-assertions peerDependencies -> acorn

this is also visible in package-lock.json

    "node_modules/acorn-import-assertions": {
      "version": "1.7.6",
      "resolved": "...",
      "integrity": "...",
      "dev": true,
      "peerDependencies": {
        "acorn": "^8"
      }
    },

but in the snyk-nodejs-lockfile-parser deptree, the peerDependencies field is missing

{
  labels: { scope: 'dev' },
  name: 'acorn-import-assertions',
  version: '1.7.6',
  resolved: '...',
  integrity: '...',
  nameVersion: '[email protected]'
}

related #104

details: peerDependencies are NOT listed in yarn.lock

acorn-import-assertions@^1.7.6:
  version "1.7.6"
  resolved "..."
  integrity ...

webpack@^5.53.0:
  version "5.53.0"
  resolved "..."
  integrity ...
  dependencies:
    ...
    acorn "^8.4.1"
    acorn-import-assertions "^1.7.6"

details: peerDependencies in pnpm-lock.yaml

  /webpack/[email protected]:
    ...
    dependencies:
      ...
      acorn: 8.5.0
      acorn-import-assertions: [email protected]

  /acorn-import-assertions/[email protected]:
    ...
    peerDependencies:
      acorn: ^8
    dependencies:
      acorn: 8.5.0
    dev: true

details: when are peerDependencies installed?

npm versions 1, 2, and 7 will automatically install peerDependencies if they are not explicitly depended upon higher in the dependency tree. For npm versions 3 through 6, you will receive a warning that the peerDependency is not installed instead.

https://nodejs.org/en/blog/npm/peer-dependencies/

As of npm v7, peerDependencies are installed by default.

https://docs.npmjs.com/cli/v7/configuring-npm/package-json#peerdependencies

... except its marked "optional" in peerDependenciesMeta

[Feature Request] Add support for integrity and resolved properties for dependency

lockfile-lint needs support for yarn v2 lockfile as mentioned in lirantal/lockfile-lint#101 .Since this repo already has good support for Yarn v2 , we would like to use this repo in order to eliminate all in-house lockfile parsing logic. However, lockfile-lint requires integrity and resolved properties for dependencies as well in order to perform current and any future potential checks.

I would like to know if the authors would be willing to accept a PR to add support for these properties.

/cc @lirantal

NPM package-lock.json TypeError: Cannot read properties of undefined (reading 'packages')

I have an older npm project (still using legacy-peer-deps) on which Snyk is failing to monitor and test. I've tracked it down to the 1.1243.0 release, which contained an upgrade of this library from 1.52.1 to 1.52.5.

The message I'm seeing (I'll gist the entire debug output at the bottom) is:

  snyk:run-test Error running test {
  error: TypeError: Cannot read properties of undefined (reading 'packages')
      at resolvedToWorkspace (/snapshot/snyk/dist/cli/webpack:/snyk/node_modules/snyk-nodejs-lockfile-parser/dist/dep-graph-builders/npm-lock-v2/index.js:111:1)
      at getChildNode (/snapshot/snyk/dist/cli/webpack:/snyk/node_modules/snyk-nodejs-lockfile-parser/dist/dep-graph-builders/npm-lock-v2/index.js:122:1)

This project doesn't have a workspaces element, and I think the issue is linked to this updated workspaces parsing:

v1.52.1...v1.52.5#diff-81f70fb04fd56113a23046eb84b767b2c22af26fb791de3e071b1ecce7170fceR205-R213

This change pulls from similar code within npm itself:

https://github.com/npm/map-workspaces/blob/ff82968a3dbb78659fb7febfce4841bf58c514de/lib/index.js#L27-L41

Which is slightly safer than the snyk/nodejs-lockfile-parser implementation. The npm code sets workspaces to [] if it doesn't exist, earlier on in the call hierarchy. If you're accessing a property of an array which doesn't exist, there's no issue. The problematic element of the Snyk implementation seems to be the pkgs['']['workspaces']['packages'] in the Array.isArray check. ['workspaces'] doesn't exist in this instance, which is fine. But then attempting to access ['packages'] breaks (recall the above error message Cannot read properties of undefined (reading 'packages')).

Strangely, I can't reproduce this on other projects we have that use a v3 lockfile, even if they don't have the workspaces element. I also can't share this project directly as it's proprietary, but I'll gist the debug information below. Let me know if you need any further information from me!

Full debug log:
https://gist.github.com/matthewarmand/3421b38ed6e1f78912cf797401527242

npm install --save-exact @magic-libraries/gql

https://github.com/magic/entities/blob/0e0ecfb1fcb2ae9e13c4ffac6dd35f0fb406f9d1/package-lock.json

Yarn 2 workspaces: private flag is no longer required

Current implementation of getYarnWorkspaces function verifies that private flag is set in package.json before analysing workspaces property. This was indeed requried for yarn v1, but no longer required for yarn v2: https://yarnpkg.com/features/workspaces

yarn2 and node 8 support

Yarn 2 support was removed because Yarn 2 requires node>10

It is not necessary to depend on @yarnpkg/core to parse the lockfile. The Yarn 2 lockfile is yaml and should be parseable by yaml. The yaml dependency only requires node>6. It seems the main issue would be forking parseDescriptor and parseRange into nodejs-lockfile-parser.

What is the appetite for a PR modifying #85 to bring back yarn2 support in a way that also supports node 8?

[🐛] yarn 2 lock file parsing issues

node -v: 12
npm -v: yarn 2
snyk -v: snyk/actions/node@master
OS: linux
Command run: snyk/actions/node@master

Expected behaviour

Work well with yarn 2 lock files

Actual behaviour

yarn.lock parsing failed with an error: Unknown token: { line: 3, col: 2, type: 'INVALID', value: undefined } 3:2 in lockfile

Steps to reproduce

create a project with yarn 2 and a github actions workflows that uses the snyk/actions/node@master action

Found issues

You use @yarnpkg/lockfile to parse the lock file, but it doesn't support yarn 2 lock files as they contain

__metadata:
  version: 4

which the parser does not support.

Debug log

  snyk test { _: [ [Circular] ], debug: true } +0ms
  snyk Error running test { error: { InvalidUserInputError: yarn.lock parsing failed with an error: Unknown token: { line: 3, col: 2, type: 'INVALID', value: undefined } 3:2 in lockfile
    at YarnLockParser.parseLockFile (/snapshot/snyk/node_modules/snyk-nodejs-lockfile-parser/lib/parsers/yarn-lock-parse.ts:65:13)
    at Object.<anonymous> (/snapshot/snyk/node_modules/snyk-nodejs-lockfile-parser/lib/index.ts:67:45)
    at Generator.next (<anonymous>)
    at __awaiter (/snapshot/snyk/node_modules/tslib/tslib.js:113:75)
    at new Promise (<anonymous>)
    at Object.__awaiter (/snapshot/snyk/node_modules/tslib/tslib.js:109:16)
    at Object.buildDepTree (/snapshot/snyk/node_modules/snyk-nodejs-lockfile-parser/dist/index.js:20:20)
    at Object.parse (/snapshot/snyk/dist/lib/plugins/nodejs-plugin/npm-lock-parser.js:43:31)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:189:7) code: 422, name: 'InvalidUserInputError' } } +0ms
Error: 
Testing /app...

yarn.lock parsing failed with an error: Unknown token: { line: 3, col: 2, type: 'INVALID', value: undefined } 3:2 in lockfile
    at test (/snapshot/snyk/dist/cli/commands/test/index.js:173:23)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:189:7)
    at Function.Module.runMain (pkg/prelude/bootstrap.js:1317:13)
    at startup (bootstrap_node.js:240:16)
    at bootstrap_node.js:661:3
  snyk analytics { args: [ { debug: true, org: undefined, showVulnPaths: 'some' } ],
  command: 'bad-command',
  org: undefined,
  metadata: 
   { local: [ true, true ],
     'generating-node-dependency-tree': { lockFile: true, targetFile: 'yarn.lock' },
     'error-message': '\nTesting /app...\n\nyarn.lock parsing failed with an error: Unknown token: { line: 3, col: 2, type: \'INVALID\', value: undefined } 3:2 in lockfile',
     error: 'Error: \nTesting /app...\n\nyarn.lock parsing failed with an error: Unknown token: { line: 3, col: 2, type: \'INVALID\', value: undefined } 3:2 in lockfile\n    at test (/snapshot/snyk/dist/cli/commands/test/index.js:173:23)\n    at <anonymous>\n    at process._tickCallback (internal/process/next_tick.js:189:7)\n    at Function.Module.runMain (pkg/prelude/bootstrap.js:1317:13)\n    at startup (bootstrap_node.js:240:16)\n    at bootstrap_node.js:661:3',
     'error-code': 422,
     command: 'test' },
  version: '1.305.0',
  os: 'Linux 4.15',
  nodeVersion: 'v8.16.2',
  id: 'e8ed2f726c2573943ce493d9d7a6f5fa898b48d1',
  ci: false,
  durationMs: 44 } +0ms

  snyk Exit code: 2 +0ms
  snyk sending request to: https://snyk.io/api/v1/analytics/cli +0ms
  snyk request body size: 990 +0ms
  snyk gzipped request body size: 550 +0ms
  snyk not using proxy +1ms

JS

https://github.com/metabase/metabase/blob/master/package.json

Add pnpm support?

We recently switched our package manager from npm to pnpm since it reduce our install time by multiple minutes in our monorepo. Since then our Snyk scans are failing, as we should have expected. We hacked something to generate a package-lock from the pnpm-lock.yaml but we'd like to have a more robust solution.

Can we start working on a PR to add pnpm support? I saw that you already have a parser for yarn that uses yaml and I think we could reuse some logic.

Thanks!

Error: Cannot find module 'debug'

HI, while trying the package at https://npm.runkit.com/snyk-nodejs-lockfile-parser I got an error:

It seems the debug package is used here but it is not explicitly declared as a dependency inside https://github.com/snyk/nodejs-lockfile-parser/blob/master/package.json.

feat: tree walker

every tree needs a tree walker ; )

npm/logical-tree

npm/logical-tree fork

i use this to build a deep node_modules with symlinks, just like pnpm

its useful to have an array of parent nodes in the walker function

sample

  const deptree = await buildDepTree(
    read('package.json'),
    read(lockfilePath),
    false,
    lockfileTypeOfName[path.basename(lockfilePath)],
    true,
  );

  function walk_deptree(_this, enter, _seen, deptreePath = []) {

    // note: this will deduplicate deps
    if (!_seen) { _seen = new Set() }
    if (_seen.has(_this)) { return }
    _seen.add(_this)

    enter(_this, function recurse() {
      for (let key in _this.dependencies) {
        walk_deptree(_this.dependencies[key], enter, _seen, deptreePath.concat([_this]))
      }
    }, deptreePath)
  }

  //deptree.forEach((dep, recurse, deptreePath) => {
  walk_deptree(deptree, function enter(dep, recurse, deptreePath) {
    // enter
    console.log(`${deptreePath.map(_ => `+ `).join('')}${dep.name}@${dep.version}`);
    recurse();
    // leave
  });

Provide usage examples

Hi there, thanks for providing this package which makes it much easier to read the various lockfile formats.

From the README I was unsure how to use this package (do I npm-install it?), and what functions I could call once it's imported. From a look in /lib/index.ts I assume I'm looking for buildDepTreeFromFiles.

It would also be helpful to have each exposed function and its parameters (does manifest mean like the package.json file?) and return types documented, or at minimum a usage example of buildDepTreeFromFiles on the README. I think I've got it figured out now, but it'd be much smoother with some docs up-front. Thanks!

yarn 2 `resolutions` causes `OutOfSyncError`

As part of the optimization of the depTree in #93, we added the following check:

nodejs-lockfile-parser/lib/parsers/lock-parser-base.ts

Lines 346 to 348 in 0bd6a0a

    
           if (!depMap[depName]) { 
        
             throw new OutOfSyncError(depName, this.type); 
        
           }

In the case of resolutions in the package.json in Yarn 2, this error will get thrown because of the way Yarn 2 puts custom resolutions in yarn.lock.

Reproduction Steps

Create a new Yarn 2 project with package.json:

{
  "dependencies": {
    "lodash": "4.17.19"
  },
  "resolutions": {
    "lodash": "4.17.20"
  }
}

Run yarn, which will create a yarn.lock:

"lodash@npm:4.17.20":
  version: 4.17.20
  resolution: "lodash@npm:4.17.20"
  checksum: b31afa09739b7292a88ec49ffdb2fcaeb41f690def010f7a067eeedffece32da6b6847bfe4d38a77e6f41778b9b2bca75eeab91209936518173271f0b69376ea
  languageName: node
  linkType: hard

"root-workspace-0b6124@workspace:.":
  version: 0.0.0-use.local
  resolution: "root-workspace-0b6124@workspace:."
  dependencies:
    lodash: 4.17.19
  languageName: unknown
  linkType: soft

Run snyk test
See error Dependency [email protected] was not found in undefined. Your package.json and undefined are probably out of sync. Please run "undefined" and try again. (the undefined is just because yarn2 is missing from lib/errors/out-of-sync-error.ts.

Explanation
In the above yarn.lock, the root workspace has a dependency on [email protected], which does not appear elsewhere in the yarn.lock because the only actual lodash is 4.17.20, caused by the resolution. This breaks the depMap check, above, even when --strict-out-of-sync=false is passed.

Potential Solutions

When --strict-out-of-sync=false is passed, ignore dependencies that are not in the depMap; this will unblock consumers who use resolutions, but it doesn't add full support for resolutions.
Figure out how to add full support for resolutions, potentially using functionality provided by @yarnpkg/core.