thescox-incomm/snyk-api-import

⚡️Snyk API powered import tool to help you automate & monitor a large scale import into Snyk organizations. Designed for onboarding with a built in queue & retries 📈

---

This repository is in maintenance mode, no new features are being developed. Bug & security fixes will continue to be delivered. Open source contributions are welcome for small features & fixes (no breaking changes)

Snyk helps you find, fix and monitor for known vulnerabilities in your dependencies, both on an ad hoc basis and as part of your CI (Build) system.

snyk-api-import

Snyk API project importer. This script is intended to help import projects into Snyk with a controlled pace utilizing available Snyk APIs.

What does it offer?

• rate limiting handling - the script will pace requests to avoid rate limiting from Github/Gitlab/Bitbucket etc and to provide a stable import.
• queue - requests to Snyk are queued to reduce failures.
• retries - the script will kick off an import in batches, wait for completion and then keep going. Any failed requests will be retried before they are considered a failure and logged.

If you need to adjust concurrency you can stop the script, change the concurrency variable and start again. It will skip previous repos/targets that have been requested for import.

Table of Contents

• Installation

• Usage

• FAQ

• Utilities

  • Creating orgs in Snyk
  • Generating import data
  • Mirroring Github.com/Github Enterprise organizations & repos in Snyk
  • Mirroring Gitlab organizations & repos in Snyk
  • Mirroring Bitbucket Server organizations & repos in Snyk
  • Mirroring Bitbucket Cloud organizations & repos in Snyk

• Contributing

• Kicking off an import

• Sync: detecting changes in monitored repos and updating Snyk projects

• Example workflows

  • AWS automation example

Installation

snyk-api-import CLI can be installed through multiple channels.

Standalone executables (macOS, Linux, Windows)

Use GitHub Releases to download a standalone executable of Snyk CLI for your platform.

More installation methods

Install with npm or Yarn

Install with npm or Yarn

Snyk snyk-api-import CLI is available as an npm package. If you have Node.js installed locally, you can install it by running:

npm install snyk-api-import@latest -g

or if you are using Yarn:

yarn global add snyk-api-import

Usage

By default the import command will run if no command specified.

• import - kick off a an API powered import of repos/targets into existing Snyk orgs defined in import configuration file. 100% support available for all project types supported via Import API.
• help - show help & all available commands and their options
• orgs:data - util generate data required to create Orgs via API.
• orgs:create - util to create the Orgs in Snyk based on data file generated with orgs:data command.
• import:data - util to generate data required to kick off an import.
• list:imported - util to generate data to help skip previously imported targets during import.

The logs can be explored using Bunyan CLI

FAQ

Error: ENFILE: file table overflow, open or Error: EMFILE, too many open files

If you see these errors then you may need to bump ulimit to allow more open file operations. In order to keep the operations more performant tool logs as soon as it is convenient rather than wait until very end of a loop and log a huge data structure. This means depending on number of concurrent imports set the tool may exceed the system default ulimit.

Some of these resources may help you bump the ulimit:

• ss64.com
• StackOverflow
• blog.mact.me

ERROR: HttpError: request to https://github.private.com failed, reason: self signed certificate in certificate chain

If your Github / Gitlab / Bitbucket / Azure is using a self signed certificate, you can configure snyk-api-import to use this certificate when calling the HTTPS APIs.

export NODE_EXTRA_CA_CERTS=./path-to-ca

Does this work with brokered integrations?

Yes. because we reuse the existing integration with your SCM (git) repository to perform the imports, the brokered connection will be used when configured.

What is supported for import command?

snyk-api-import supports 100% of the same integration types and project sources as the Import API documentation. If an example is not in the docs for your use case please see the API documentation